Multi-Purpose Interactive Cognitive Platform

ABSTRACT

An interactive cognitive platform is provided that uses image-based interactivities for diagnosis, treatment, and to analyze the progress (for disease and/or treatment) of cognitive diseases via a graphical user interface. The interactivities engage multiple cognitive domains and involve Gestalt principles—aspects which can be personalized to be more effective for each user. The cognitive platform can be used by healthcare workers to produce diagnostics or treatment plans for specific cognitive conditions and diseases with a cognitive component. The cognitive platform can be used for gaming, stress reduction and skills development and performance enhancement for those without cognitive problems.

PRIORITY CLAIM AND CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority benefit of U.S. Provisional PatentApplication No. 62/721,665 (Docket # CY-6-2), entitled “MULTI-PURPOSEINTERACTIVE COGNITIVE PLATFORM,” filed on Aug. 23, 2018, by TamiEllison;

this application is a continuation-in-part of U.S. patent applicationSer. No. 16/427,305 (Docket # CY-6-7), entitled “SYSTEM AND METHOD FORCREATING AN IMAGE AND/OR AUTOMATICALLY INTERPRETING IMAGES,” filed onMay 30, 2019, which in turn is a continuation-in-part of U.S. patentapplication Ser. No. 16/262,884 (Docket Number CY-6-4), “SYSTEM ANDMETHOD FOR CREATING AN IMAGE AND/OR AUTOMATICALLY INTERPRETING IMAGES”by TAMI ROBYN ELLISON, filed on Jan. 30, 2019; which claims prioritybenefit of U.S. Provisional Patent Application No. 62/626,208 (Docket #CY-6-1), entitled “SYSTEM AND METHOD FOR IDENTIFYING CONTIGUITYCHARACTERISTICS IN AN IMAGE,” filed on Feb. 5, 2018, by Tami Ellison,which is incorporated herein by reference; and also claims prioritybenefit of U.S. Provisional Patent Application No. 62/721,665 (Docket #CY-6-2), entitled “MULTI-PURPOSE INTERACTIVE COGNITIVE PLATFORM,” filedon Aug. 23, 2018, by Tami Ellison, which is incorporated herein byreference; U.S. patent application Ser. No. 16/262,884 (Docket NumberCY-6-4), “SYSTEM AND METHOD FOR CREATING AN IMAGE AND/OR AUTOMATICALLYINTERPRETING IMAGES” by TAMI ROBYN ELLISON, filed on Jan. 30, 2019 isalso a continuation-in-part of U.S. patent application Ser. No.15/884,565 (Docket # CY-6-3) entitled “SYSTEM AND METHOD FOR GENERATINGCOMPOSITE IMAGES,” filed on Jan. 31, 2018, by Tami Ellison, which claimspriority benefit of U.S. Provisional Patent Application No. 62/499,655(Docket # CY-6-5), entitled “PHOTAGE 2.5D-METHOD AND SYSTEM FOR CREATINGDYNAMIC VISUAL ILLUSIONS USING COMPLEX, JUXTAPOSED AMBIGUOUS IMAGES,”filed on Feb. 1, 2017, by Tami Robyn Ellison; U.S. patent applicationSer. No. 16/427,305 (Docket # CY-6-7), entitled “SYSTEM AND METHOD FORCREATING AN IMAGE AND/OR AUTOMATICALLY INTERPRETING IMAGES,” filed onMay 30, 2019, is a continuation-in-part of U.S. patent application Ser.No. 15/884,565 (Docket # CY-6-3) entitled “SYSTEM AND METHOD FORGENERATING COMPOSITE IMAGES,” filed on Jan. 31, 2018, by Tami Ellison,which claims priority benefit of U.S. Provisional Patent Application No.62/499,655 (Docket # CY-6-5), entitled “PHOTAGE 2.5D—METHOD AND SYSTEMFOR CREATING DYNAMIC VISUAL ILLUSIONS USING COMPLEX, JUXTAPOSEDAMBIGUOUS IMAGES,” filed on Feb. 1, 2017, by Tami Robyn Ellison; U.S.patent application Ser. No. 16/427,305 (Docket # CY-6-7), entitled“SYSTEM AND METHOD FOR CREATING AN IMAGE AND/OR AUTOMATICALLYINTERPRETING IMAGES,” filed on May 30, 2019, claims priority benefit ofU.S. Provisional Patent Application No. 62/721,665 (Docket # CY-6-2),entitled “MULTI-PURPOSE INTERACTIVE COGNITIVE PLATFORM,” filed on Aug.23, 2018, by Tami Ellison;

this application is also a continuation-in-part of U.S. patentapplication Ser. No. 15/884,565 (Docket # CY-6-3), entitled “SYSTEM ANDMETHOD FOR GENERATING COMPOSITE IMAGES,” filed on Jan. 31, 2018, by TamiEllison, which is incorporated herein by reference; U.S. patentapplication Ser. No. 15/884,565 (Docket # CY-6-3) claims prioritybenefit of U.S. Provisional Patent Application No. 62/499,655 (Docket #CY-6-5), entitled “PHOTAGE 2.5D-METHOD AND SYSTEM FOR CREATING DYNAMICVISUAL ILLUSIONS USING COMPLEX, JUXTAPOSED AMBIGUOUS IMAGES,” filed onFeb. 1, 2017, by Tami Robyn Ellison. The contents of all of the abovelisted applications are incorporated herein by reference, in theirentirety

FIELD

This specification generally relates to a multi-purpose interactivecognitive platform.

BACKGROUND

The subject matter discussed in the background section should not beassumed to be prior art merely as a result of its mention in thebackground section. Similarly, a problem and the understanding of thecauses of a problem mentioned in the background section or associatedwith the subject matter of the background section should not be assumedto have been previously recognized in the prior art. The subject matterin the background section may merely represent different approaches,which in and of themselves may also be inventions.

Cognitive issues affect hundreds of millions of people around the world.Current neurocognitive evaluations are based on decades/century-oldsiloed skills' tests using simple stimuli Today's “multi-domain”assessments are individual skills assessments compiled into batteries.Not surprisingly, these evaluations are limited by ceiling and flooreffects, and lack the sensitivity to detect subtle changes over time,delaying early detection, diagnosis and interventions. Despitetremendous gains in knowledge/technology, there is a lack ofnon-invasive, objective, quantifiable, authentic multi-domain assessmenttools and training products to support brain health and fitness.

Cognitive platforms can be used for a variety of reasons includingtherapy diagnosis and treatment of cognitive disorders, gaming, and evenin the field of artificial intelligence. However, new and improvedcognitive platforms able to address deficiencies and limitations ofprevailing platforms are needed.

BRIEF DESCRIPTION OF THE FIGURES

In the following drawings like reference numbers are used to refer tolike elements. Although the following figures depict various examples ofthe invention, the invention is not limited to the examples depicted inthe figures.

FIG. 1 is a block diagram of an example of a system that analyzes animage for a multi-purpose interactive cognitive platform.

FIG. 2 is a block diagram of an embodiment of the architecture of themachine system of FIG. 1.

FIG. 3 shows an example of entity relationship diagrams of an embodimentof a database schema of the system of FIGS. 1 and 2.

FIG. 4 shows an example of a flowchart for performing a contiguityanalysis of an image for a multi-purpose interactive cognitive platform.

FIG. 5 shows an example of flowchart of an embodiment of a method forcomputing parameters associated with contiguities and/or contiguitylines for a multi-purpose interactive cognitive platform.

FIGS. 6A and B show an example of a flowchart of a method of computingcontiguity continuity values using a stitched image for a multi-purposeinteractive cognitive platform.

FIG. 7 shows an example of a flowchart of a method of storing images anddate to a library for use with an interactive cognitive platform.

FIG. 8 shows an example of a flowchart of a method to build a userprofile for a multi-purpose interactive cognitive platform (a method ofOn-boarding).

FIG. 9 shows an example of a flowchart of a method of using amulti-purpose interactive cognitive platform for a returning user (amulti-session protocol).

FIG. 10 is an example of a flowchart showing three options for how aregistered user may interact with an interactive cognitive platform.

FIG. 11 is an example of protocol options for a user starting with theselection of one or more images from a graphical user interface.

FIGS. 12A and B are an example of a method of making a user interactiveworkspace.

FIG. 13 is an example of a method of a user interacting with a cognitiveplatform to generate a metric and/or update a user skill level.

FIG. 14 is a second example of a method for interacting with a cognitiveplatform by a user (see also FIG. 13).

FIG. 15 is an example of a collaborative method in which professionalusers (e.g., health-care workers) analyze data from users based on skilllevels.

FIG. 16 is a second example of a collaborative method in whichprofessional users (e.g, health-care workers) analyze data from usersbased on skill levels (see also FIG. 16).

FIG. 17 is an example of a method that allows professional users tocreate a cognitive platform for specific uses (e.g., tests, diagnoses,treatments of specific diseases) in a collaborative way.

FIG. 18 shows an example of a Graphical User Interface (GUI) PICSSiprototype for a multi-purpose interactive cognitive platform.

FIGS. 19A-19D show an example of Rules that can be used for measuringcontiguity ranges (19A), color block depth (19B), spatial colorcontiguity (19C), and ambiguity factor (19D) for a multi-purposeinteractive cognitive platform.

FIGS. 20A and B show an example of the application of quadrant-basedmeasures, a stitch-based angle determination for analysis of an image.

FIGS. 21A-C show an example of the application of the use of colorthresholds to extract contiguities for image analysis.

FIGS. 22A-D show an example of composite images, including multi-stable(22A and 22B) and stable figure-ground relationships (22C and 22D).

FIGS. 23A-F show examples of two-image composite images where thecontiguity has been serially removed, stabilizing the figure-groundrelationship.

FIGS. 24A-C show examples of two-image composite images that showhierarchical relationships in the figure-ground positioning.

FIG. 25 is a flowchart showing an embodiment of a portion of cognitiveinteractivity platform for creating an interactivity.

FIG. 26 shows a flowchart of a method of interacting with aninteractivity.

DETAILED DESCRIPTION

Although various embodiments are described in this specification mayhave been motivated by various deficiencies with the prior art, whichmay be discussed or alluded to in one or more places in thespecification, the embodiments of the invention do not necessarilyaddress any of these deficiencies. In other words, different embodimentsof the invention may address different deficiencies that may bediscussed in the specification. Some embodiments may only partiallyaddress some deficiencies or just one deficiency that may be discussedin the specification, and some embodiments may not address any of thesedeficiencies.

The flowchart and block diagrams in the FIGS. illustrate thearchitecture, functionality, and operations of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

In this specification, the term “logic” refers to a specialized circuit,embedded software, middleware, (note embedded software is hardware andmiddleware includes hardware), software, a specialized processor, a VeryLarge Scale Integration (VLSI) chip, a configured Application SpecificIntegrated Circuit (ASIC), a configured Field Programmable Gate Array(FPGA), or other logic circuit optimized and/or configured for the taskin question (see U.S. Pat. No. 6,785,872 for methods for convertingalgorithms into circuits, which is incorporated herein by reference).

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

FIG. 1 is a block diagram of an example of system 100 that analyzes animage for an interactive cognitive platform. System 100 may includemachine system 101, which has processor system 102, input system 104,output system 106, memory system 108, image database 110, communicationinterface 112, and third party system 114 and third party database 116.In other embodiments, System 100 may include additional componentsand/or may not include all of the components listed above.

System 100 relates to a platform in which cognition and cognitiveprocesses can be applied to people and/or machine processes, whichutilize, and/or which are modeled on, human cognitive and visionprocesses. In support of brain health, the platform can be applied topeople across the cognitive spectrum towards supporting cognitivefunction, information and language processing, learning, training,screening, stimulation, skills development, stress reduction, therapy,and remediation purposes. Cognition can be viewed in terms of individualbrain and neurological processes as well as holistically considering thetotality of conscious and subconscious input and/or stimuli and theinterpretation, analysis, storage and translation of such inputs into awide range of output forms. The platform's image-based interactivitiesmay be used to support human perception, cognition, behavior andinteractions with the environment and the world, directly and/orindirectly, such as through a secondary device or other type ofinterface. The secondary device may be worn, implanted or transmittedsignal, in the immediate, short-term and/or for later retrieval.

Cognition can be viewed in terms of allocated cognitive domainsresponsible for critical functions and processes, including: memory,attention, executive functions, language and communication,sensorimotor, and visual-spatial operations. Each of these domains mayinclude a multiplicity of processes and skills. Each of the processeswithin a domain and/or multiple domains may be integrated with one ormore other processes and/or domains with crossover relationships.Neurocognitive functions provide a framework for how the brain functionsand/or a gateway to understanding brain dysfunctions.

Brain processes associated with learning and by default memory andattention operations, among other cognitive domains, can also bedescribed in terms of thinking skills. Thinking skills are traditionallyhyphenated into “higher” order thinking skills (HOT) and “lower” orderthinking (LOT) skills; a taxonomy first developed by B. S. Bloom in the1950's and generally applied to educational targets. The principlesrelated to thinking can be generalized to learning where processesrequire the mustering of both higher order thinking skills includingcritical, logical, reflective, metacognitive, and creative thinking foranalyzing, evaluating, synthesizing and creating, together with lowerorder thinking (LOT) skills which include: applying, understanding andremembering. The totality of higher and lower order thinking skillsbecome integrated when considering cognition and cognitive processes andtheir participation in perception. HOT and LOT skills can be framed bythe domains described previously with the application of the processeswithin the cognitive framework to help people interact with theirenvironment and the world around them. More recently, the fields ofneuroscience and educational practices has gained an appreciation forlearning styles—a differentiation which not only applies to learning andtraining, but also to assessments and how these are conducted tointegrate individual learning styles, including: auditory, verbal(linguistic) visual-spatial, kinesthetic (movement), among others.Individuals can have a bias toward a particular learning style, butgenerally display a mix of learning styles which can be differentiallymanifested depending on task requirements.

Different learning styles may be accommodated in a variety of ways. Inembodiments of the platform, a visually driven system (e.g., with visualsensory input), provide triggers for visual memory—an important driverand/or indicator of cognition. Further, the associative connections tovisual memories and pattern recognition can be used to integratedifferent learning styles. Learning in this case not defined by booksbut rather by inputs and associating neural connections. From a purelearning standpoint, instructions are provided in text, images, and/oraudio format. The assessments integrate verbal descriptive responses;kinesthetics (moving puzzle piece parts). Rewards for task completionmay be in the form of audio output, and may include background music toprovide audio lifts (e.g., music commonly associated with a positiveoutcome and/or a victory). The platform may allow users to expressresponses and/or demonstrate capabilities, incorporate non-visualcontent, and expression, which may support a mix of learning styles.When designing interactivities and companion tests/assessments bothprocesses are relevant—the activity and assessments (or described astests in academic settings).

Brain processes, whether in healthy and/or impaired individuals, can befurther framed within the context of top-down processes and bottom-upprocesses. In bottom-up processing sensory input can be received,assembled and integrated through multiple steps; whereas, in top-downprocessing cognition draws on, uses and applies models, ideas, andexpectations (inferences) to interpret sensory data and generally leadsto some kind of output and/or response. Sensory input and the upstreamand/or downstream processes, including analysis can be complex. Forexample, recognizing the nuances of fire can mean discriminating betweena building which is on fire, an out-of-control conflagration versus asingle candle burning on dining room table or knowing that a pot on astove is potentially hot (an associated connection to fire). Associativeneuronal connections to concepts and practices all require some kind ofsensory input and the integration of multiple sensory inputs, (seeingthe fire, feeling the heat, smelling smoke), prior knowledge and memoryinputs (top-down inputs) and woven together in a rich web ofconnections.

Sensory input can include: visual, auditory, tactile, motor/kinestheticmovement, gustatory among other types of inputs which can be senseddirectly and/or indirectly or transduced through a secondary mediumand/or device, including an implantable or wearable, as well as throughcomputer-brain/neural interfaces and other human-machine interfaces,whether through permanent or temporary interactions.

A significant body of research has been dedicated to understandingcognitive function as it relates to brain health, well-being, reasoning,decision-making, learning styles, skills development in healthyindividuals and in those with changes in brain health associated withdisease conditions. A diversity of processes, changes, differences andimpacts, and/or altered states which are reflected in the range ofdiseases and conditions with overlapping symptoms and impacts on one ormore cognitive processes. Conditions with a cognitive component,include: ADHD, ADD, Autism, Multiple Sclerosis, Parkinson's Disease,Type II Diabetes, Atrial Fibrillation, aging/mild cognitive impairment,Alzheimer's disease, and Dementias, stress, Chemotherapy,post-anesthesia cognitive dysfunction, schizophrenia, among othertransient, progressive, acute and/or chronic physiological,psychological neuromuscular and other conditions.

The platform described here is designed to support brain health throughdiagnostic assessment, intervention, and treatment modalities; and, toengage cognition in support of learning and skills development andtraining enhancements as a standalone methodology delivered through theplatform, and/or in conjunction with other assessment tools, devicesand/or therapies, such as exercise equipment and/or with a passiveand/or active exercise protocols including whole body vibration,transcranial magnetic processes, and/or as an adjunct modality, and/orassessments to support cognitive well-being and cognitive processes.

The platform can be used as part of a system to help support brainhealth as a potential treatment modality, an intervention that can bedelivered as a device-based intervention using smart devices, such as acomputer, tablet, phone or other type of interfacing device. For examplethe interfacing device may be a hands-on interactive or in view-onlymode; and/or delivered offline as either a hands-on manipulative and/orin view-only mode. Offline, the platform materials and interactive toolscan be projected and/or printed on a pre-sectioned substrate or asubstrate, which can be sectioned, allowing the parts to be manipulated(e.g., as a picture puzzle that needs to be assembled); or, which havebeen printed or transferred onto a different medium, and/or arepresented in view-only mode printed and/or projected on a substrate;and/or a hybrid of online and offline components. The platform may beinclude a subset of overlapping assessments which can be conducted withboth the device-based and offline tools for crossover multi-modalanalysis and tracking. The platform can provide a distinction betweenverbal and non-verbal users; and with use cases including minimallyconscious individuals who can access the view-only options. In theseembodiments, assessments of interactions requires the use of biometricssuch as eye-tracking and EEG as an index of engagement.

The platform provides a method for developing a treatment plan for apatient, or for delivering a multiplicity of interactivities,interventions and/or user engagements according to healthcare workers(e.g., clinicians, researchers) or other user and/or system protocols tomeet and/or address individual and/or group cognitive and/or trainingrequirements for healthy individuals as well as for those who areexperiencing cognitive challenges in order to address individualcognitive domains a part of holistically engaging multiple cognitivedomain processes and skills as an integrated system. An interactivity isan activity that the user (e.g., a patient) participates in, as part ofinteracting with the platform. Interactivities include games, puzzles,therapeutic exercises, diagnostic tests, for example. A user can be anyone of the following: a patient, an individual, a healthcare worker, aresearcher, a professional gamer, game maker, and/or a clinician, forexample. In some embodiments, the term “user” can refer to any one orall of the above. In any of the embodiments, each of the terms apatient, an individual, a healthcare worker, a researcher, aprofessional gamer, game maker, and/or a clinician may be substitutedone for another to obtain a different embodiment. In some embodiments,the term healthcare worker can refer to any worker in the healthcareindustry including, but not limited to, a researcher, a doctor, aclinician, therapist, a nurse, and a laboratory technician.

System 100 provides a multi-purpose interactive cognitive platform forcognitive well-being and skills training, and assessment/diagnosis ofcognitive dysfunction. System 100 provides a platform for healthcareworkers to implement assessments for a variety of cognitive functionsand dysfunctions.

System 100 is a network of systems including multiple machinescommunicating via a network, which may be used for treatment anddiagnosis, for example, by analyzing images and/or creating artisticimages by combining multiple images into one image, such as byinterweaving multiple images with one another. The image sets may embedmultiple Gestalt principles (figure-ground, closure, continuation),engaging top-down cognition and bottom-up sensory processing, as usersvirtually reassemble the spatially separated image parts in virtuallyreconstructing the intact image.

In one embodiment, the cognitive platform is a PICSSi/Mem+ prototypewhich is designed to engage cognition with a simple question, such as“what do you see?” The term “Mem+” refers to testing for memory plusother aspects of cognition related to memory. The PICSSi/Mem+ systemuses real-world images—enriched visual stimuli—to cooperatively engageglobal cognition (skills and processes across multiple cognitivedomains). Using real-world images combined with the simple question,improves the quantity and quality of captured data (as compared to ifother images and/or questions were used), allowing for directmeasurements of overall cognitive status as well as domain-specifictask/skill metrics, towards developing sensitive, reliable cognitivetools. In an embodiment, the question is open ended. In an embodiment,the question is one that only requires a one word or one phraseresponse. In an embodiment, the question is 7 or less words. In anembodiment, the question is 10 or less words. In an embodiment, thequestion is 15 or less words. In an embodiment, the question is 20 orless words. In an embodiment, the question requires that the useranalyze interweaved image sets, focusing a range of cognitive in theprocess, including language and memory domains, but also attention,visual spatial and executive functions processes and skills. Insituations where some users may not have firsthand experience with thecontent of an image, for example, a field of sunflowers, but the userhas experienced flowers, the image set can still be of value intraining, treatment and assessment. Similarly, while lakes are familiarto a significant number of people, even those who have never experienceda lake can recognize a lake in its relationship to water and/or a bodyof water.

In one embodiment, the platform can be deployed in and through a devicewith components on a tablet, computer, phone, television, smart deviceand/or other virtual, augmented and/or mixed reality device and/ormedium as part of the Internet of Things (IoT) ecosystem ofinterconnected devices. The interactive components can be used inhands-on and/or hands-free and/or view-only mode. The hands-on mode mayinclude manipulatives with multiple types of input devices includingtouch-screens, mouse, stylus, pads, and/or Tangible User Interfaceprops, virtual projections, voice commands, and/or other types of inputdevices. The hands-free modality may include multiple types ofinterfaces, including neural feedback, eye-tracking, biofeedback,sensors implanted into the human body, sensors attached to the humanbody wearable, and/or other types of add-on system and/or device, orother biometrics tools. The input may include wifi (e.g., via a radiofrequency local area network), infrared, ultraviolet, low frequencysound, ultrasound, and/or Bluetooth, for example.

In an embodiment, a tangible user interface (TUI) is a user interface inwhich the user interacts with digital information through the physicalenvironment. The TUI gives physical form to digital information, and mayinclude sensor to sense the manipulation of physical objects andmaterials other than a keyboard. In an embodiment, a TUI of thisspecification does not include a mouse (although, in this embodiment,mouse input may be used, without a TUI, instead a TUI and/or in additionto a TUI, a mouse is not included in the scope of the term TUI of thisspecification). The TUI prop provides a tactile interface, givingdigital information, such as digital puzzle pieces, a physical form. TheTUI prop transforms digital information in manipulatable and perceptibleparts of the platform. TUI props, within the Internet of Things space,can be embedded with additional sensors to capture otherwiseinaccessible user data through traditional active surface devices orother types of inputs. In an embodiment, a TUI may include a physicalrepresentation that is computationally coupled to underlying digitalinformation, such as images and text. In an embodiment, a TUI includesspace-multiplex both input and output, concurrent access and/ormanipulation of interface components, specific devices (via which inputis sensed); spatially aware computational devices; and/or spatiallyreconfigurable of device.

In one embodiment, the platform includes offline interactive componentswhich can be delivered visually through printed matter, including, butnot limited to: paper, plastic, glass and/or wood substrates. Theoffline components can include manipulatives where images are printed insections on wood substrates. In one embodiment, each component image isprinted on four (4) 14 cm×3 cm substrates, for a total single composedpicture of 14 cm×12 cm in size. Different sized manipulatives can beprinted based on the substrate used, including varying the width of thesections, and number of sections, including half and quarter-sizedsections and smaller. In one embodiment, individual image manipulativescan be printed on a chip-board substrate and cut accordingly, or aprinted image can be section and mounted onto a substrate rather thanbeing printed, transferred and/or sublimated onto a substrate, or usesnap-together sections which can be split and/or combined together. Inone embodiment, the hybrid system would include the use of a TUI prop.In an embodiment, the prop's digital display surface would show an imagesection or image part which would be “released” to an active surfacewhen the user correctly places the image part, displayed on the propsurface, proximal to the mapped game board surface.

In one embodiment, the platform can include all of its integratedcomponents, including: image library, image sets, image database whichincludes: integrated software, delivery and server-side storage,interactivities, skill levels, interactivity progressions algorithms,complexity values, composite values, user interfaces, user datatracking, real-time feedback, data logging, assessments, reporting andalert tools to provide users and/or professionals with one or moremetrics of cognitive status. In one embodiment, the platform can also berepresented as multiple modules which can be interchanged and/orconfigured to meet individual and group requirements according toclinical health specifications.

Machine system 101 includes one or more machines that run an imageanalysis system. Each machine of machine system 101 may run thecognitive platform/image analysis system independently and/or as adistributed system. Machine system 101 may include one or more Internetservers, network servers, a system for analyzing images, may include oneor more mobile machines and/or may include other machines that includemachine vision, for example.

In at least one embodiment, in machine system 101, each image and/oreach image of a plurality of images may be analyzed to identifycontiguity characteristics in the image that facilitate identificationof visual qualities and characteristics indicative of how the viewerobserves the image for use in treatment and/or diagnosis of cognitiveissues. In an embodiment, a contiguity is a continuous region havingrelatively uniform and identifiable color and content characteristics,and which may span the entire width of an image or a portion of it. Inan embodiment, a contiguity is a region that is recognized by the systemas one region.

As an aside, the value of a color may be represented asHue-Saturation-Value instead of by wavelength of light. The pixel valuesmay be used to represent the Hue-Saturation-Value or the color.Alternatively, each color may be represented by a separate pixel value.Returning to the discussion of uniformity, in another embodiment, acolor is considered uniform if the variation of the pixel valuerepresenting the color varies by less than 10%, less than 5%, or lessthan 1% (depending on the embodiment). In another embodiment, a color isconsidered uniform if the variation of the pixel value representing thecolor varies by 10% or less, 5% or less, or 1% or less (depending on theembodiment). In another embodiment, a color is considered uniform if thevariation of the pixel value representing the color varies by no morethan 25 bits, no more than 15 bits, no more than 5 bits, no more than 3bits, or no more than 2 bits (depending on the embodiment).

In an embodiment, contiguities (which may be referred to as “contigs”)may be compared to, horizon-like edges according to certaincharacteristics and may include horizontal edges that are associatedwith a horizon. However contiguities may also contain significantly moreinformation beyond just information about edges. A contiguity may be anygenerally horizontal feature, such as a line or a block of pixels thatare within a predetermined threshold of uniformity of color betweenpixels that are within a predetermined number of pixels or distance formone another (thereby having a “local uniformity”). Local uniformityrefers to the uniformity in color between nearby and/or neighboringpixels. In an embodiment, contiguities extend for at least half thewidth of the image. Contiguities may be associated with a multiplicityof characteristics within a given image and any given contiguity may beassociated relationships between that contiguity and other contiguitiesthat are in the same components of the image and as conveyed in acomposite image. Contiguity characteristics include: contiguity number,contiguity stacking, linearity, continuity, angularity, depth/saliency,regularity, and color composition. Contiguities may be framed by theircontent, color and context information.

In an embodiment, the contiguities that are of interest are those thatextend horizontally across the image, which for example extend at least75% of the width of the image (in other embodiments smaller or largerpercentages of the width may be used). In an embodiment, thecontiguities of interest can make an angle of 45 degrees or less with ahorizontal line (in other embodiments the angle may be 75 degrees orless, 60 degrees or less, 30 degrees or less, or 15 degrees or less, forexample). A contiguity can separate regions of the image and/or maydefine a region of the image. In at least one embodiment, the contiguitycharacteristics may include contiguity lines that separate differentcolor segments in the image, e.g. the contiguities may form edgesbetween the color segments. A contiguity line may separate a contiguityfrom other regions. In at least one embodiment, the images displaylandscape scenes in which the contiguity lines are naturally occurringhorizon edges, horizon type edges, and/or border lines (e.g., edges thatextend more than 50% of the width of the image and that are at an angleof less than 45 degrees). In an embodiment a contiguity line may also behorizontal. For example, in urban settings contiguity lines can behorizontal, but which depends on the subject matter. The edges of thecontiguity may separate color sections of the image, for example theedges of a contiguity may separate between the background and theforeground, between objects, between different parts of a background,between different parts of a foreground, different parts of an object,and/or the like.

The contiguity characteristics may enable a person viewing the image tomentally organize parts of the scene displayed in the image intodifferent areas that allow the viewer to understand what is shown, andcan be used to train a computer vision system to recognize continuitieseven between disrupted contiguities, which may be absent or obstructed.The terms disrupt and disruptor are used interchangeably with the termsdistract and distractor. Either may be substituted one for the other toobtain different embodiments. The contiguity lines can provide acontrast, enabling the person's brain or the computer vision system toorganize and evaluate the image and to resolve ambiguities in the image,image set, and/or image scene. In at least one embodiment, contiguitiesmay be used to inform image classification (that is may be at least onefactor used in determining the classification of an image) and can beused to identify content and aid in finding objects and/or regions inthe image. The classification of an image is at least a part ofidentifying the content of the image. A classification system may havecategories and subcategories and the smallest subcategories may beobjects or parts of objects that are identified.

In at least one embodiment, contiguity may be defined and used to trainsystems to recognize parts of a whole. For example, a contiguity maycorrespond to (and thereby identifying the contiguity identifies) asingle object or a contiguity may correspond to (and thereby identifyingthe contiguity identifies) a distinctive part of an object. Whentraining a machine, contiguities may need to be identified in bothsingle images as well as composites, and in composite images thecontiguities may be split (or divided) by the other images of thecomposite image. A composite image is an image formed by combining atleast two images together. For example, the at least two images may beinterweaved with one another. The figure and ground relationships in acomposite image is another value vis-a-vis training sets that may beused to further define relationships of objects in an image. An element,object, or region of an image is in the figure position when the elementobject or region is located where a main character of photograph wouldbe located. An element, object, or region is in the ground position ifthe element, object, or region forms a contiguity that stretches acrossthe image.

In at least one embodiment of the cognitive platform, the user's abilityto recognize the parts of the whole, to apply a label, to virtuallyreconstruct the hyphenated mage segments, to differentially focusattention on the figure or ground positioned image in a compositemusters coordinated multi-cognitive domain engagement in resolving theambiguities inherent in the image sets based on the user's knowledge,experience and memories. In at least one embodiment, the user'sinteractions with the image sets, i.e. the composite of interweavedimage sections, is through the gamified image parts and theinteractivities mix defined by the user, a clinician, a therapist orresearcher.

As another example, two contiguities may, or contiguity lines maysection off, a region of an image that is one object or a group ofrelated objects. Contiguities may be seen as familiar horizon lines,interfaces with a known and/or predictable color, color “context,”and/or content characteristics, and may include information about thelocation of shapes and information about the density of a feature. The“context” of the color context refers to an assigned context, a contextthat is known for other reasons, a context that is predictable, and/or acontext that is probabilistically inferred. The determination of thecontext may be based on the source of the data and/or user inputspecifying the context. For example, if the data has a known context,the accuracy of identifying objects may be improved and/or facilitated.The word “density” may refer to a concentration of colors or to thesaliency of elements within a defined space which may have additionalcontext, optionally, as a result of the co-localization of the elementswithin a given context to help in its identification and/or correctplacement of the image part as the users works through the platform'sinteractivities. For example, the interface with a vertically positionedblue of relatively uniform density is likely to be a sky. A dark elementon the surface or at the interface is likely to be a ship—all based onknown contexts, associations and references that were previously learnedover time.

As a further example, bodies of water often form contiguities and areregions of high density of water droplets. As another example, colorblocks may aid in the identification of objects or regions contained inan image or a plurality of images or image scene. The context may aid ininterpreting whether a contiguity is water. Water is transparent, butreflects the colors around it—a stormy sea with dark clouds will havevery different characteristics than a calm sea or lake reflecting a bluesky with still water. Nonetheless, based on the context both can stillbe recognized and/or inferred as a body of water.

In at least one embodiment, the system 101 may be configured to identifythe contiguity lines by applying various image processing filters to theimage, e.g. Sobel, thresholding, and/or the like, to identify thecontiguities in the image. In at least one embodiment, the system can beconfigured to perform a stitch analysis of the image to designate thecontiguity characteristics that are preferred for use for analyzingcomponents in the image and to facilitate identifying images withsimilar or overlapping characteristics.

Contiguities may be analyzed by juxtaposing non-adjacent image segmentsand masking a portion of the image in the process, which provides arapid snapshot of the symmetrical and/or asymmetrical, color differencesand contiguity regularity and continuity in deriving Aesthetic andAmbiguity Ratings towards developing a Compositing Factor for an image(such as by using a 1:3 stitch). Stitching may involve removing (ormasking) portions of an image. For example, vertical sections of theimage may be removed or masked. Throughout the specification the terms“remove” and “mask” and their conjugations, when used in reference toremoving or masking part of an image are used interchangeably.Throughout the specification, the terms “remove” and “mask” and theirconjugations may be substituted one for another to obtain differentembodiments. The vertical sections removed may be of the same size asone another and equally spaced from one another. The juxtaposition andmasking may be part of a stitching method where an image may be dividedinto 3 sections and section 1 is juxtaposed next to section 3. Section 2may be masked in the process and gradually revealed as the stitchedimage is peeled. Measurements may be taken of the combination of one ormore contiguities based on the complexity of the images comprising theimage sets for image analysis purposes (FIG. 20A-E show an example ofthe a stitch and peel process). For example, the system can beconfigured to identify and designate contiguity lines that arehorizontal, vertical, within a predetermined degree of angle deviationand/or the like, according to predetermined parameters provided to thesystem. Peeling or backstitching refers to putting back parts of theimage that were masked or removed. In at least one embodiment, thestitch analysis may enable the system to identify contiguitycharacteristics that are obstructed by objects in the image that segmentthe contiguity line. In at least one embodiment, the stitch analysis maybe implemented by dividing the image into a predetermined number ofsections, e.g., three sections. At least one of the sections can bemanipulated, e.g. shifted, to mask or overlap another section in theimage. The overlapping section can then be peeled off the masked sectionto reveal portions of the masked section such that the contiguity linecan be identified from the portions of the image being revealed via thepeeling. An abrupt change in pixel value or Hue-Saturation-Value (HSV)in regions of the stitched image may indicate a potential disruption inthe contiguity making the region a target region for further evaluation.A minimal change (within predetermined thresholds/limits) in pixeluniformity or a progression along a hue spectrum in other regions of thecontiguity represents continuity of the contiguity across the width ofthe image.

In at least one embodiment, the system can be configured to identify thecontiguity lines by applying various image processing filters to theimage, e.g., via a Sobel filter, thresholding, and/or the like, toidentify the contiguities in the image. In at least one embodiment, thesystem can be configured to perform a stitch analysis of the image todesignate the contiguity characteristics that are preferred for use foranalyzing components in the image and to facilitate identifying imageswith similar or overlapping characteristics. For example, the system canbe configured to identify and designate contiguity lines that arehorizontal, vertical, within a predetermined degree of angle deviationand/or the like, according to predetermined parameters provided to thesystem. In at least one embodiment, the stitch analysis can enable thesystem to identify contiguity characteristics that are obstructed byobjects in the image that segment the contiguity line. In at least oneembodiment, the stitch analysis can be implemented by dividing the imageinto a predetermined number of sections, e.g. three sections. At leastone of the sections can be manipulated, e.g. shifted, to mask or overlapone other section in the image. The overlapping section can then bepeeled off the masked section to reveal portions of the masked sectionsuch that the contiguity line can be identified from the portions of theimage being revealed via the peeling.

The identification and analyses of contiguities in images is used toassign contiguity characteristics, complexity values and figure-groundspecifications as these relate to the image itself and its relationshipto other images when combined as a composite with one or more additionalimages.

Processor system 102 may include any one of, some of, any combinationof, or all of multiple parallel processors, a single processor, a systemof processors having one or more central processors and/or one or morespecialized processors dedicated to specific tasks.

Input system 104 may include any one of, some of, any combination of, orall of a keyboard system, a mouse system, a trackball system, a trackpad system, buttons on a handheld system, a scanner system, a microphonesystem, a connection to a sound system, and/or a connection and/orinterface system to a computer system, intranet, and/or internet (e.g.,IrDA, USB), for example Input system 104 may include a graphical userinterface that third parties can interact with.

Output system 106 may include any one of, some of, any combination of,or all of a display, a monitor system, a handheld display system, aprinter system, a speaker system, a connection or interface system to asound system, an interface system to peripheral devices and/or aconnection and/or interface system to a computer system, intranet,and/or internet, for example. Output system 106 may include a networkinterface via which third parties interact with machine system 101.Input system 104 and output system 106 may be the same system ordifferent system.

Memory system 108 may include, for example, any one of, some of, anycombination of, or all of a long-term storage system, such as a harddrive; a short-term storage system, such as random access memory; aremovable storage system, such as a floppy drive or a removable drive;and/or flash memory. Memory system 108 may include one or moremachine-readable mediums that may store a variety of different types ofinformation. The term machine-readable medium is used to refer to anynon-transient medium capable carrying information that is readable by amachine. One example of a machine-readable medium is a non-transientcomputer-readable medium. Another example of a machine-readable mediumis paper having holes that are detected that trigger differentmechanical, electrical, and/or logic responses. Memory system 108 maystore one or more images for users to select from and/or that users mayuse.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device havinginstructions recorded thereon, and any suitable combination of theforegoing. A computer readable storage medium, as used herein, is not tobe construed as being transitory signals per se, such as radio waves orother freely propagating electromagnetic waves, electromagnetic wavespropagating through a waveguide or other transmission media (e.g., lightpulses passing through a fiber-optic cable), or electrical signalstransmitted through a wire. Rather, the computer readable storage mediumis a non-transient (i.e., not-volatile) medium.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server.

In the latter scenario, the remote computer may be connected to theuser's computer through any type of network, including a local areanetwork (LAN) or a wide area network (WAN), or the connection may bemade to an external computer (for example, through the Internet using anInternet Service Provider). In some embodiments, electronic circuitryincluding, for example, programmable logic circuitry, field-programmablegate arrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention. Aspectsof the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

Image database 110 may be a database of images that may be analyzed,that were analyzed, and/or from which composite images may be formed.Optionally image 110 may include a relational database. Optionally,image database 110 may associate with images and/or portions of an imageattributes, such as contiguity, ambiguity, juxtaposition (which israting of a contiguity, which will be discussed further below), a colormap and/or other color properties, saliency, complexity, aestheticvalue, edge information, context information, content and/or categorydescription, spatial information about contiguities, and/or thresholdinformation. Optionally, image database 110 may be associated with adatabase server for retrieving information from image database 110.Optionally, the image server (if present) may be a relational databaseand the database server may be executed by processor system 102 or byits own processor system.

Communication interface 112 is an interface, via which communicationsare sent to and from machine system 101. Communications interface 112may be part of input system 104 and/or output system 106.

Third party system 114 is a third party system and interacts withmachine systems 101 to analyze images. Third party system 114 mayinclude third party database 116, which stored images of the third partysystem 114. Third party system 114 is optional.

Processor system 102 may be communicatively linked input system 104,output system 106, memory system 108, and communication interface 112.Processor system 102 may be communicatively linked via any one of, someof, any combination of, or all of electrical cables, fiber optic cables,and/or means of sending signals through air or water (e.g. wirelesscommunications), or the like. Some examples of means of sending signalsthrough air and/or water include systems for transmittingelectromagnetic waves such as infrared and/or radio waves and/or systemsfor sending sound waves.

In at least one embodiment, machine system 101 may be configured toimplement the platform and/or receive an image from third party system114, for example. The image may be stored in the image database 108,which may store other images. Processor system 102 may retrieve, and/orthe image may be provided, image to processor system 102 for thecontiguity analysis. Processor system 102 may implement a user interfacefor testing user, providing therapy to users, analyzing tests taken byusers, plan and or construct a therapy and/or test and/or trainingregimen for a user. In at least one embodiment, machine system 101 maybe configured to size and crop the image to a predetermined size and/orto divide the image into sections and each section may be sized andcropped. The cropping may remove portions of the image or the portionsof the image that are not wanted, or edges of the image that cause theimage to be too large for generating the composite image, and/or tocentralize dominant contiguities and color blocks in the image or in aportion of an image. In at least one embodiment, machine system 101 canbe configured to generate an image grid map. The image grid map may begenerated, for example, by designating the Cartesian coordinate systemto the image designating numerical coordinates of the image. In at leastone embodiment, the numerical coordinates may be pixel locations of theimage or may be used to construct (and/or define) quadrants,sub-quadrants and/or some other predetermined areas of the image.

FIG. 2 is a block diagram of the architecture 200 of machine system 101,which may be designed to analyze an image and/or create composite imagesfor an interactive cognitive platform. Architecture 200 may includestitching logic 202, ambiguity logic 204, and saliency logic 206,contiguity logic 208, edge identification logic 210, and color map logic212, region/grid generator 214, processor system 216, aesthetic valuecalculator 218, complexity calculator 220, juxtaposition calculator 222,the artificial intelligence logic 224, thresholding logic 226, sizingand cropping logic 228, patient interface 229, healthcare workerinterface 230, health assessment logic 232 is the logic that correlatesthe performance and progress of a user with the user's health assessment232, test assessment logic 234, and test adjustment logic 236. In otherembodiments, architecture 200 may include additional components and/ormay not include all of the components listed above.

Stitching logic 202 performs the stitching of an image. During thestitching a portion of an image (e.g., one or more horizontal strips)may be removed from the image. After removing the portions of the image,the image may be analyzed, such as by computing the contiguity, andoptionally other characteristics of the image, such as the saliency,color block depth, ambiguity, color map, edge detection, color thresholdmap, brightness and/or threshold map. After removing the portions of theimage, and analyzing the image, the portions may be returned. After eachportion of the image is restored, the image is again analyzed todetermine contiguities, determine contiguity characteristics, perform amulti-contiguity analysis, and optionally determine othercharacteristics.

Ambiguity logic 204 determines the ambiguity of an image and/or of aportion of an image. The ambiguity is a measure of the degree to whichthere are elements that may have multiple interpretations.

Saliency logic 206 computes the saliency of an object, image, or portionof an image. The saliency is a measure of the contrast within andbetween objects or elements. Specifically, the saliency is a measure ofinternal contrast. Regions of high saliency may be regions that includea foreground type object. In other words, if the saliency is above apredetermined threshold value the saliency may be one or one of multiplefactors used to determine whether a region is a foreground object orpart of a foreground object. Alternatively, the saliency value may bepart of a formula for determining whether a region is part of aforeground object.

Contiguity logic 208 identifies contiguities in an image and/orcontiguity lines in an object. Contiguity lines may aid in identifyingseparate regions that have different meaning from one another, such asseparating land from sky, foreground from background, street frombuildings, plains from mountains or hills.

Edge identification logic 210 may identify edges in an image. In anembodiment, edge identification logic may divide images into regionsthat have pixels with brightness values above and below a particularthreshold and/or have a wavelength of color within a particular window,to help identify regions in the image. Edge identification logic 210 mayalso divide regions that are below a particular color threshold. Colormap logic 212 maps the color of different regions. The image may beseparated out into images of different colors and color maps of theimage may be constructed (e.g., a blue image made from the blue pixelsof the image, a red image made from the red pixels of the image and agreen image made from the green pixels of an image.

Region/grid generator 214 may generate a grid and/or divide the imageinto multiple regions (e.g., quadrants, halves, thirds, eighths), whichmay be further divided into sub-regions. The regions, subregions, andgrid may be used to identify the locations of elements in an image.Processor system 216 may be an embodiment of processor system 102, andmay be capable of implementing a stitching analysis, determiningcontiguities, computing aesthetic value, complexity, and/orjuxtaposition of an image and/or portions of an image.

Artificial intelligence logic 224 may be a neural network or otherartificial intelligence logic. Artificial intelligence logic 224 mayreceive a training set of images, and/or stitched images that areassociated with the contiguity values, an identification ofcontiguities, an identification of contiguity lines, an aesthetic value,a complexity value, and/or juxtaposition values, and an identificationof objects and/or of object parts in the image. After receiving thetraining set, artificial intelligence logic 224 may be trained toidentify objects based on the stitched images that are associated withthe contiguity values, an identification of contiguities, anidentification of contiguity lines, an aesthetic value, a complexityvalue, and/or juxtaposition values, for example. Thresholding logic 226creates a derived image by setting all pixels above a threshold to onevalue and below the threshold to another value, which may be helpful inidentifying edges and/or other features. Thresholding logic 226 isoptional and may be part of edge identification logic 210. Sizing andcropping logic 228 may automatically size and crop the image or portionsof the image.

Patient interface 229 is the interface via which the patient (whom maybe referred to as a patient user) interacts with the system 200. Patientinterface 229 may be used by the patient for taking assessments, whichmay be in the forms of game, for measuring cognitive ability.Alternatively or additionally, patient interface 229 may be used forproviding therapy to the patient (or other user).

Healthcare worker interface 230 is the interface via which thehealthcare worker interacts with the system 200 for collaborating withother healthcare workers, reviewing test results and/or progress ofpatients, and/or for assigning assessment and/or therapy to patients.

Health assessment logic 232 is the logic that correlates the performanceand progress of a user with the user's health. Health assessment logic232 may assess cognitive abilities of a patient and/or progress of apatient in response to a therapeutic treatment (the treatment may be inthe form of games played on system 200 by the patient and/or othertreatments). Health assessment logic 232 may be based on previousperformances by the user, how the user's performance compares with otherpatients at the same difficulty level, how the user's performancecompares with the general public,

Test Assessment Logic 234 is the logic that assesses the performance andprogress of a patient user in taking a particular test or group oftests.

Test Adjustment Logic 236 is the logic that adjusts the difficulty ofthe test based on the user's skill level. In some embodiments, thedifficulty may need to be increases due to the user becoming morecomfortable with the test (becoming an “expert” on the test). In someembodiments, the test difficulty may be reduced because the user is notgetting enough of the answers right. In some embodiments, if the user isnot getting 50%, 40%, 30% 25%, 20%, 15%, 10%, 5% or less right, and/orif the patient is taking a longer time, by a predetermined threshold(e.g., by 1, 1.5, 2, 2.5, 3, or 3.5 standard deviations) than mostpatients, for example, the difficulty level is adjusted down. In atleast one embodiment, speed and/or accuracy thresholds may be set basedon one or more of the following: gender, age, and known cognitive healthstatus. In some embodiments, if the user is getting more than 50%, 40%,30% 25%, 20%, 15%, 10%, 5% right, and/or if the patient is taking a lesstime than an average person by a predetermined threshold (e.g., by 1,1.5, 2, 2.5, 3, or 3.5 standard deviations) the difficulty level isadjusted up.

FIG. 3 shows an example of entity relationship diagrams of an embodimentof a database schema 300 of the system of FIGS. 1 and 2. Database schema300 may include an image table 302, a stitched image table 304, an imageelement table 306, a relationship image table 308, and threshold map310. In other embodiments, database schema 300 may include additionalcomponents (such as tables) and/or may not include all of the components(e.g., tables) listed above.

Image table 302 may include various attributes associated with theimage. A particular object in a table may be found by searching theattributes of the object. For example, a user may find a particularimage by searching for an image having a particular set of attributes.For example, image table 302 may include among its attributes an imageidentifier, category identifier, a saliency value, and a contiguityrating value (or juxtaposition value), edge map, and/or other attributessuch as content and/or color preferences. Image table 302 may alsoinclude an edge value, which may be generated by an edge identificationtable. The image identifier is a primary key and a unique identifier ofan image.

Each of the stitched image table 304, an image element table 306, arelationship image table 308, and threshold map 310, have the imageidentifier as a key, so that each threshold map, image relation, imageelement may be associated with one image. The stitched image table 304lists each stitched image of each image. Each image may have multiplestitched images. The attributes of the stitched image table 304 mayinclude the image identifier, stitched image identifier, map ofcontiguities, stitched image contiguities, saliency value, ambiguityvalue, edge map, and other attributes. The image identifier identifiesthe image that the stitched image was generated from, and the stitchedimage identifier uniquely identifies the stitched image. Stitched imagetable 304 may also include a type, which describes the type of stitch,which may indicate how much of the image was removed and/or the portionremoved. The saliency, ambiguity, and edge map may be the saliencyvalue, ambiguity, and edge map of the stitched image.

Image element table 306 may be a table of elements identified in images.Image element table 306 includes an image identifier identifying whichimage the element was found in, and an element identifier identifyingthe element. Image element table 306 includes an image identifier,relationship identifier, stitched identifier, type of element, textdescription, and/or other attributes. Image element table 306 mayinclude a descriptor that identifies any relationship that involves theelement. Image element table 306 may include a type of element thatdescribes the type of element.

Relationship table 308 may be a table of relationships identified inimages. Relationship table 308 includes an image identifier,relationship identifier, stitched identifier, type of relations, textdescription, number of elements and other elements. The image identifieridentifies which image the relationship was found in, and therelationship identifier uniquely identifies the relationship.Relationship table 308 may include a descriptor that identifies anyobjects in the image that are related by the relationship.

Threshold map table 310 may be a table that lists all the thresholdmaps. The attributes of threshold table 310 may include a relationshipidentifier, stitch identifier, type of threshold, threshold value,threshold map. The image identifier identifies the image from which thethreshold map was created, and a threshold map identifier identifies thethreshold map. The type of threshold indicates the type threshold, suchas whether the threshold map is a black and white threshold map or colorthreshold map. Threshold attribute is the value used as the thresholdfor making the threshold map.

FIGS. 6-7 provide methods for performing contiguity analyses for use incognitive platforms. Images can be analyzed using manual and/orautomated methods to derive Ambiguity and Aesthetic values based on oneor more dominant image contiguities. Contiguity analysis can beconducted for both online and offline components of the platform as theimages can be printed from digital files. The digital files can be usedby the platform and/or transferred to a medium or a substrate such aspaper, metal, and wood. Data based on a manual analysis of contiguitycharacteristics can be entered manually into the digital platform orusing a hybrid system of manually entering data using a series ofprompted fields with a stylus or other marking device, and where thedata is attached to the image and used to calculate Ambiguity, Aestheticand Compositing Factor values for each image (and Saliency values as asubset of Ambiguity Factors (AF1, AF2, AF5, AF6). The automation of theprocess facilitates deriving the Compositing Factor for 2- and 3-imagecomposited images whether as part of the digital platform or if usedwith offline components in assigning skill levels to image sets.

FIG. 4 shows an example of a flowchart 400 for performing a contiguityanalysis of an image for use in interactive cognitive platforms. In step402, method 400 starts. For example, in step 402, one or more images arereceived, retrieved, captured, taken and/or formed, via processor system102 and/or communication interface 112.

In step 404, the image may be sized and cropped (step 404 is optional),via processor 112 and/or sizing and cropping logic 228. In other words,the image may be enlarged or reduced and/or edges may be removed byprocessor 112 and/or sizing and cropping logic 228. In at least oneembodiment, machine system 101 may be configured to size and crop theimage to a predetermined size. The cropping may remove portions of theimage that are not wanted, or edges of the image that cause the image tobe too large for generating the composite image, and to centralizedominant contiguities and color blocks.

In step 406, a quadrant map and an image grid map are generated, viaregion/grid generator 214. In at least one embodiment, machine system101, via region/grid generator 214, may generate a quadrant map, whichcan equally divide the image into quadrants spanning the entire area ofthe image (or into another number of regions, such as halves, thirds,fifths, sixths, eighths, etc. In at least one embodiment, the quadrantscan be arranged along a Cartesian coordinate system including an X-axisand a Y-axis, in which the center of the Cartesian coordinate system canbe predetermined according to predetermined parameters, such as positionof dominant content, color blocks, and/or the like. The dominant contentmay be content that occupies either a majority of the image or a greaterportion of the image than other content identified. For example, asingle contiguity that is larger than all other contiguities may be thedominant content. In other embodiments, other coordinate systems may beused, such as polar coordinates, hyperbolic coordinates, ellipticalcoordinates, etc.

In at least one embodiment, machine system 101, via region/gridgenerator 214, may be configured to generate the image grid map. Theimage grid map can be generated, for example, by designating theCartesian coordinate system to the image designating numericalcoordinates of the image. In at least one embodiment, the numericalcoordinates can be pixel locations of the image or can be used toconstruct quadrants or some other predetermined areas of the image. Thecoordinates generated by region/grid generator 214 may be the pixelcoordinates or may be the pixel coordinate plus (or minus) an additiveconstant and multiplied (or divided) by a scaling factor. In at leastone embodiment, machine system 101, via region/grid generator 214, isconfigured to generate a measurement area within the image grid map. Themeasurement area may be designated as a predetermined area of the imagegrid map in which the contiguity characteristics may be identified. Inat least one embodiment, the measurement area enables identification ofobjects in the image.

In step 408, the contiguities of the image are analyzed, via contiguitylogic 208. In at least one embodiment, machine system 101, viacontiguity logic 208, is configured to analyze the image to identifycontiguities in the image. In at least one embodiment, the contiguity ofthe image can include contiguity lines, e.g. the edges that separatedifferent regions of the image according to color differences betweenthe areas, color combinations, and/or the like. The identification ofthe contiguities may be performed by identifying edges and/or regionshaving a uniform coloring and/or brightness (within a predeterminedthreshold). In at least one embodiment, the contiguities can enable aviewer of the image to identify objects, backgrounds, foregrounds, orthe like in the image. The contiguities may appear in differentlocations within the image according to the visual content of the image,image set, or image scene comprised of at least one image. Optionally,the contiguities are identified, via contiguity logic 208, prior toperforming any of the substeps of step 408. Contiguity logic 208 maycall edge identification logic 210 and/or thresholding logic 226 toassist in identifying contiguities.

In step 410, one or more images are stitched, via stitching logic 202,by removing one more parts of the image. Optionally, the parts removedmay be rectangular sections stretching from the top of the image thebottom of the image. For example, the middle third of the image may beremoved.

In step 412, the contiguities of the stitched image are identifiedand/or analyzed, by contiguity logic 208. Contiguity logic 208 may callstitching logic 202 to facilitate identifying contiguities. Thestitching may further facilitate determining contiguities (that were notpreviously identified) and determining objects that interfere with thecontiguity, breaking up the contiguities. Color blocks that have similarcolor but different colors may create object interference—interferencethat make it difficult to distinguish the border between two or moreobjects. Stitching and peeling (via stitching logic 202 and/orcontiguity logic 208) may facilitate identifying two separatecontiguities and/or separate objects despite the object interference andmay help bracket the location of a border between where two colorregions and/or two objects. In at least one embodiment, the stitchanalysis may include masking and progressively peeling portions of theimage to enable analyzing a reduced portion of the image to enabledefining contiguity characteristics, e.g. contiguity lines, horizonlines, interfaces breaking up the lines, linearities, continuities,regularities, object locations, for example. The steps for angularities,stitching and peeling are discussed further below.

In step 414, a determination is made whether predetermined criteria aremet indicating to backstitch the image. For example, in an embodiment, adetermination may be made whether the image has been backstitched and,if the image has not been backstitched, whether the image should bebackstitched. In another embodiment, the user may enter input thatindicates whether to backstitch the image, and if it is determined thatthe input indicates that the user wants the backstitching to beperformed, then it is determined that the backstitching is desired. Ifit is desired to backstitch, the method proceeds to step 416. In step416 the image is backstitched. Optionally, each time step 416 isperformed a fraction of the image that was previously removed (ormasked) is put back into the image (or unmasked). After step 416, themethod returns to step 412, where the backstitched image analyzed (e.g.,for contiguities). Steps 412, 414, and 416 may be performed multipletimes, until all of the backstitching desired is performed.

In at least one embodiment, machine system 101, can be configured toperform the serial backstitch to an image, set of images, or a scenewithin an image. The serial backstitch may compress the contiguity edgeanalysis by arranging in an adjacent manner the non-adjacent sections ofan image. The serial backstitch can be configured to compress the imageon which the contiguity and/or edge analysis is performed by bringingtogether non-adjacent sections of the image.

Returning to step 414, if all the backstitching needed has beenperformed, the method proceeds to step 418. In step 418, thecomputations of the multiple implementations of step 416 are combined.For example, the values representing the contiguity characteristics thatwere determined in each backstitch are averaged by the total numberbackstitching steps 416 were performed. The backstitching and evaluationof contiguities is discussed further below.

In step 420, an image contiguity rating (“CR”) value (ambiguity value,or juxtaposition value) is stored in association with the image. In thisspecification the terms juxtaposition value and contiguity rating valueand ambiguity value are used interchangeably. Throughout thisspecification either term may be substituted for the other term toobtain different embodiments. The locations of the contiguities are alsostored in association with the data, for further analysis of the image.In at least one embodiment, machine system 101 can be configured tostore the image CR value. The image CR value can include a rating thatenables machine system 101 to determine an image compatibility for usein generating the composite images. Composite images may be acombination of multiple images. For example, two or more images may beinterwoven with one another to form a composite image. The image CRvalue may be based on multiple parameters, such as the definiteness ofthe contiguity in the image (e.g., how much contrast exists between thecontiguity and surrounding regions), the number of contiguitiesidentified in the image, spatial distribution of the contiguities, thewidth of the contiguities, the color composition of the contiguities,and/or the angularity of the contiguity (that is, the angularity is theangle at which contiguity is oriented—a larger angle between thehorizontal axis and the contiguity may detract from the contiguity andtherefore lower the CR, in a convention in which a higher CR valuerepresents more contiguities with a higher distinctiveness of individualcontiguities, where viewed in isolation of the other contiguities).

FIG. 5 schematically illustrates a method 500 for generating acontiguity rating value and other related parameters for use incognitive platforms. In step 502, dominant contiguities are identifiedby edge identification logic 210. In at least one embodiment, machinesystem 101 is configured to identify dominant contiguities. The dominantcontiguities can be identified, for example, implementing Sobel filtersto the image, or another edge identification method, and then using theedges to determine the size and distinctiveness of each contiguity. Thedominant contiguities can be determined by the edges of the image aswell as the color blocks in the image. For example, each contiguity maybe assigned a score. In an embodiment, a contiguity that includes adominant edge is a dominant contiguity. Dominant edges are dominantcontiguities, but not all dominant contiguities may not be dominantedges as a contiguity can also be a color block.

Continuing with the description of step 502, in step 502, the totalnumber of contiguities and dominant edges are also identified in theimage. In an embodiment, a dominant edge is an edge that extends acrossat least a majority of the image. In an embodiment, a dominant edge isan edge that is longer than the majority of other edges. In anembodiment, a dominant edge is an edge that is longer than the majorityof edges and extends more horizontally than vertically, and/or extendsdiagonally. In an embodiment, a dominant edge-type contiguity wouldextend horizontally across 75% or more of the image. In at least oneembodiment, machine system 101 is configured to verify the total numberof contiguities, which include the dominant edges in the image, whichmay be in any direction. The dominant edge can be determined byperforming a corner and border identification of the image andidentifying edges between color blocks that are above a predeterminedcontrast and/or threshold level. A dominant edge can have a CR valuebetween 0.75-2.25. In at least one embodiment the dominantedge/contiguity is the edge/contiguity that is used for makingmeasurements, and which contributes to the image's switch capacity.Optionally, a dominant edge has a contrast between adjacent regions thatis above a predetermined threshold. For example, in an embodiment, adominant edge has a contrast of at least 8:1, at least 10:1, at least20:1, or at least 100:1.

In step 504 thresholding is performed by threshold logic 226.Thresholding logic 226 may form a binary image by setting pixels of theoriginal image above the threshold to white (or black) and the pixelsbelow the threshold being set to black (white). The threshold may be forbrightness, a particular color, and/or hue. In at least one embodiment,machine system 101, by thresholding logic 226, may be configured toapply a threshold filter function to the image. The threshold filterfunction of thresholding logic 226 may aid in partitioning the imageinto foreground and background parts. The thresholding of thresholdinglogic 226 may be based on a particular reduction of the colors in theimage. The reduction of the color in the image may be performed byrepresenting a color that is not in the color palette of the machinethat made the image with the closest color in the palette and/or adithering pattern of the close colors. The threshold filter function ofthresholding logic 226 may generate a binary image of the image toenable edge recognition or detection between the foreground, thebackground, and/or objects in the image, for example. The termsrecognition and detection are used in interchangeably throughout thespecification. Throughout this specification, each may be substitutedfor the other to obtain different embodiments. The threshold filterfunction may include computing, by thresholding logic 226, a histogram,and clustering the colors into bins and setting the threshold, so as tooperate between two clusters of bins. Thresholding logic 226 may choosethe threshold based on color, hue, or brightness level that dividesbetween colors, hues or brightnesses that are associated with differentlevels of entropy (e.g., perhaps pixels having a brightness of above 200are associated with regions having more entropy than those below thethreshold and so the binary image is formed with the threshold set at abrightness of 200). The threshold of thresholding logic 226 may be setbased on an object attribute. For example, pixels that are known to beassociated with a particular attribute or interest (e.g., an object ofinterest) tend to have a particular color or brightness and so thethreshold may be set and a color or brightness above or below thatparticular color. The threshold of thresholding logic 226 may be basedon spatial filtering. For example, certain regions of the image may beremoved from the image, prior to setting the threshold. In at least oneembodiment, a multi-level thresholding filter can be implemented bythresholding logic 226 to designate a separate threshold for each of thered, green, and blue components of the image, which can then becombined, for example. Alternatively, multiple brightness thresholds maybe set by thresholding logic 226 to produce multiple binary images.

In step 506, thresholding logic 226 may generate a threshold-spatial map(which may be referred to as a T-spatial map). The threshold spatial mapstores the locations (e.g., the pixel coordinates of each pixel of theoriginal image that has a value above a threshold and/or each pixel ofthe original image that has a pixel blue below a threshold may be storedas the T-spatial map). In at least one embodiment, machine system 101can be configured to generate, by thresholding logic 226, the T-spatialmap, for example, by implementing a threshold filter to the image. Theapplication of the T-spatial map to an image helps define edges,contiguities, and dominant contiguities. The line in the image thatdivides between regions of the image having the pixels that are aboveand below the threshold may be and/or may be related to edges,contiguity lines, and dominant contiguities in the image. Similarly, theregions having pixels of one of the two types, may be contiguities ormay be parts of contiguities (depending on the size and shape of theregion, whether the region is identified as being part of a largerregion and/or other characteristics of the region).

In step 512, color hues are compressed, by color map logic 212. Thecompression of the colors may involve, for each pixel determining whichof a predetermined number of colors the pixel of the original image isclosest to. In at least one embodiment, machine system 101 can beconfigured to compress the color hues. The color hue compression mayreduce the colors in the image to a predetermined number of colors, forexample, to a number of colors that is within a range of 2-6 colors, forexample.

In step 514 the averaged hue percentages are computed, by color maplogic 212. For example, for each of the predetermined colors thepercentage of the total number of pixels in the image that are binnedwith (closest to) one of the predetermined colors. Thus, if one of thecolors (e.g., red) has 2500 pixels associated with that color and theimage has 1096×1096 pixels, then there are 2500*100%/(1096×1096)=0.2%red pixels. In at least one embodiment, machine system 101 can beconfigured to calculate, via color map logic 212, the averaged huepercentages. Optionally, a map is constructed having the pixel locations(e.g., pixel coordinates) of each color. The averaged hue percentages ofthe colors may be identified in the image locations.

In step 516, the hue compression (“HC”) spatial distribution is mappedby the color map logic 212. In at least one embodiment, machine system101 may be configured, by the color map logic 212, to map the huecompression spatial distribution. In other words, the probability of apixel having a particular color being in a particular region is computed(e.g., as the percentage of the pixels in a particular region havingthat color). The HC spatial distribution can be correlated to locationaccording to a higher-order probability distribution and/or correlationbetween the pixels of the image and the location of the colors in theimage. The higher order probability refers to other information that mayskew the probability distribution. For example, perhaps, as a result ofbinning the pixels, it is known that 30% of the pixels are blue.Perhaps, as a result of user input, prior images, a category to whichthe image belongs (or other information), it is expected that the imageincludes a region in the upper half of the image representing the sky,and as a result, based on prior images there, is a 90% chance of a bluepixel being located in the upper half of the image and only a 10% chancethat a blue pixel is located the lower half of the image. Then for thisimage, there is a 27% chance that pixels in the upper half of the imageare blue and 3% chance that pixels in the lower half are blue. Thelikelihood of a particular pixel being a particular color, depending onwhere the pixel is in the image, may be affected by the context,saliencies, and a knowledge reference matching pixel distribution (thatis, based on prior distributions of the pixels of prior similar images).

In step 518, a hue compression spatial map may be generated by color maplogic 212. In at least one embodiment, machine system 101 can beconfigured to generate the hue compression spatial map. The huecompression spatial map provides a mapping of the colors providedthrough the hue compression. As part of step 518, color map logic 212may compute the locations of color blocks (each color block has thecolor of the average of the color of the block or the hue with the mostpixels in its bin). Optionally, each block of a grid is overlaid on theimage and is assigned its average color as the color of that block, bycolor map logic 212.

In step 522 color blocks are compared to one another, by color map logic212. In at least one embodiment, machine system 101 can be configured,by color map 212, to compare the color blocks, which may determinedifferent color blocks in the image and may determine similarities anddissimilarities within and across the image grid map. Regions of colorblocks (where each region is a group of adjacent blocks of the samecolor) may be compared according to different quadrants in the imagegrid. The comparing of the color blocks may be in order to determine thedifferent values. For example, in a black and white image, the colorblock comparison can differentiate between colors having a binary valueof zero for white color blocks and a binary value of one for black colorblocks. In a second example, the image may include color blocks such asgreen and blue, where each color is represented by a distinct value,which enables comparing the color blocks within the image grid map.

In step 524, symmetrically-placed color blocks may be mapped by colormap logic 212. In at least one embodiment, machine system 101, by colormap logic 212, may map color blocks that have a symmetrical shape.Machine system 101, by color map logic 212, may determine that the colorblocks are symmetrical according to the pixel location or the locationwithin the grid of the color block pixels on the image grid map and mayevaluate the asymmetry of a color block, by color map logic 212. In atleast one embodiment, the number of grid boxes of the color block on theimage grid map may be compared, by color map logic 212, to determine theedges of a region having adjacent block of the same color to determinewhether the region of having a group of color blocks of the same coloris symmetric, across and within the region of the color blocks of thesame color, and may be compared to color block depth_(ST) (CBD_(ST))data obtained as being symmetrical or showing symmetrical colorcharacteristics, such as blue hues in a region of sky. The “ST” in thesubscript of the term “color block” stands for the word “stitch,” andthe number “ST” indicates the percentage of the total image that remainsafter the stitching. For example, color block depth67 means a colorblock value performed in an image that was stitched by removing ⅓ of theimage leaving ⅔ of the image and the value assigned according to rulesdescribed in FIG. 11B. The shape of the region of blocks having the samecolor may be indicative of an underlying contiguity and may place limitson the size and shape of the underlying contiguity. Using the bins, thecolor block depth may be computed. The image is divided into fourblocks, where each block is a quadrant of the image. For each quadrantthe color with the most pixels in that color's bin is determined, andthat is the “color mode” for the block (the “color mode” of a block isthe color—of the 2-6 colors into which the image is mapped that occursmost often in that block). If all four quadrants have the same colormode, the color block depth is 1. If two adjacent blocks have one colormode and the other two adjacent blocks have another color mode, thecolor block depth is 0.75. If two adjacent blocks have one color modeand the other two blocks have each have a color mode different from oneanother and different from the first two blocks, the value is 0.5. Iftwo nonadjacent blocks have one color mode and the other two nonadjacentblocks have another color, the color mode block depth is 0.5. If allquadrants have different color block modes, the color block depth has avalue of 0. If two nonadjacent blocks have one color mode and the othertwo blocks each have a color mode that is different from one another anddifferent from the first two blocks, the color block depth is 0. Eachquadrant may be further subdivided into quadrants and a color blockdepth may be computed for each quadrant. The color block depth may becomputed for different degrees of the stitched or backstitched image.

In step 526, a color block depth 100 (CBD₁₀₀) map is generated by colormap logic 212. In at least one embodiment, machine system 101 can beconfigured to generate the CBD₁₀₀ map. The image may be divided into apredetermined number of blocks. Quadrants that can be defined aspositive and negative values arranged on the Cartesian coordinate systemor with a numerical label, Q1, Q2, Q3 and Q4. The number of color blockpatterns identified by machine system 101, in each quadrant, relative toother quadrants in the image can provide a relational analysis ofdifferent color portions of the image, their distribution and symmetry,and which can be mapped onto the grid of the map to generate the CBD₁₀₀map. The nuanced differences are regions which are subjected to furtheranalysis. As quadrants are drilled down into sub-quadrants (andsub-sub-Qs) is where CB differences become more evident allowing for theidentification of IE and VD. Each quadrant may be analyzed individually,and any quadrant that has features that correspond to something ofinterest may be further divided into quadrants (or other sectors) andanalyzed individually and each sub-quadrant, having featurescorresponding something of interest may be further subdivided andanalyzed individually. The process of identifying sectors havingfeatures corresponding to something of interest and then furthersubdividing those sectors may be continued until there are two fewpixels in the sectors with which to make further analysis (e.g., wheneach sector only has one pixel).

The values for CBD₁₀₀ are based on the rules which will be described,below, in FIG. 11B. The color block map of the original intact image andthe various stitched images may be compared and the characteristics ofthe image derived from the color maps from each stitch may be averaged.

In step 528, the hue compression spatial map and CBD₁₀₀ map are combined(e.g., integrated or superimposed on one another, so that one mapappears foreground and the other map appears as background). In at leastone embodiment, machine system 101 combines the hue compression spatialmap and the CBD₁₀₀ map. The hue compression spatial map generated fromthe threshold function may be aligned with the CBD₁₀₀ map to provide aunified map for recognizing the necessary edges for designating thecontiguities in the image based on the color composition. The combinedhue compression spatial map and CBD₁₀₀ map may be used to maintain theembedded color information of the image.

In step 530, a CBD₁₀₀ is generated in at least one embodiment, machinesystem 101 can be configured to generate the CBD₁₀₀, which is thecomposited map including the overlaid information obtained by aligningthe hue compression spatial map and the CBD₁₀₀ map.

In step 532, the T-spatial map and the CBD₁₀₀ are combined. In at leastone embodiment, machine system 101 can be configured to combine (e.g.,integrate) the T-spatial map and the CBD₁₀₀.

In step 534, a contiguity number (or value) is generated by contiguitylogic 208. Color block data and spatial data may also be generated bycontiguity logic 208, as part of step 534. In at least one embodiment,as part of step 534, machine system 101 may generate the contiguitynumber, the color blocks and the spatial data. The contiguity number maybe the number of contiguities designated in the image based onpredetermined parameters (e.g., based on predetermined thresholds forthreshold maps and predetermined number of stitches and peels, arepredetermined set of bins of hue, and predetermined grid, and block sizefor the blocks of the regions of color blocks having the same color).

In step 536 an image saliency value is generated. In at least oneembodiment, machine system 101 can be configured to generate the imagesaliency value. The image saliency value provides a unique quality for agroup of pixels or for a single pixel relative to surrounding pixels andthe rest of the image, and enables easier analysis of the image. In oneembodiment, the saliency is represented by a combination of contiguityfactors including: contiguity number, number of color blocks, colorblock depth 100, and the spatial color contiguity comparison. Regionswhere color or brightness differences may be present are identified bythe differences in the distribution and the number contiguities andcolor blocks in an image.

The image saliency value sets a contour for extracting information fromthe image to enable edge detection, e.g. each pixel in a region that issimilar with respect to a predetermined characteristic or computedproperty, such as color, intensity, texture or the like. In other words,since the saliency value is an indication of whether a particular regionis of interest (e.g., as a result of having a different color,brightness, texture, and/or other characteristics than neighboringregions) if the saliency value crosses a particular threshold value theregion may be further analyzed to determine characteristics ofsub-regions with the region of interest. In this specification, thewords brightness and intensity are interchangeable, either may besubstituted for the other wherever they occur to obtain differentembodiments.

In step 538, the saliency value is stored in image database 110 and/orpassed on to other methods that make use of the saliency. The imagesaliency value, which is a measure of internal contrast, contributes tothe dominance of a subset of image characteristics defined in part orwhole by continuous and/or a contiguous group of color blocks ofrecognized elements and their corresponding juxtapositions (orContiguity Rating-CR values), or as defined by the shape of the group ofcolor blocks. As will be discussed further below, the ambiguity value isgiven by Ambi_(SAL)=Σ(AF₁+AF₂+AF₅+AF₆).

AF₁, AF₂, AF₅, and AF₆ are discussed further below, and the steps ofFIG. 5A that compute each ambiguity factor is indicated in FIG. 5A. Inan embodiment, if Ambi_(SAL)<5.5, the images contains a significantamount of nuanced or poorly defined distractions—no clear attentionfocus, save for the contiguities present in the image. Images in thiscategory can be used to focus on nuanced details as an attractor and/ordistractor element. If Ambi_(SAL) is between 5.5-14 then there is abalanced color blocking and contiguity/edge sharpness (an optimal rangefor looking at details in an image and/or for focusing on a particularobject or element in the image. If Ambi_(SAL)>14, the image contains asignificant number of discontinuous contiguities, little or no colorsymmetry, and the objects may be disrupted; there are lots of parts tolook at. Images in this category can be used to focus on nuanced detailsas attractor and/or distractor element).

FIG. 6A schematically illustrates a method 600 a of peeling, accordingto at least one embodiment for use in cognitive platforms. In step 610a, peeling operations are performed at predetermined values, such aspredetermined percentages of stitching and/or peeling. In at least oneembodiment, machine system 101 can be configured to peel a first section(e.g., a first 30% of the image), and then a second section at thepredetermined values (a second 30% of the image). Alternatively, thedifferent percentage could be used, such as 25% or 10%.

In step 612 a, irregular edges (IE) are mapped. In step 612 a, a map ofirregular edges is computed. The map may be based on the regions (e.g.,quadrants and blocks of the quadrants) of the region map, and the mapfor each region may be computed. In at least one embodiment, machinesystem 101 can be configured to map the irregular edges, which can beedges that include shapes, contrast hue, and/or color difference withthe surrounding areas. The edge irregularity may be computed bycomputing differences between edge parameters, such as the differencesin the angle, contrast, brightness, color, and hue of the edge.Differences between edge irregularities of different degrees ofstitching/peeling and/or thresholding may also be computed.

Using either the original image or the stitched image, deviations offthe X-axis relative to the dominant contiguity may be evaluated settingup a grid to define the Intrusion Area, which is the area that thevertical intrusion intrudes into an area above (and/or optionally below)the dominant contiguity. The vertical disruption by a Vertical Disruptor(VD) can be in the contiguity may be objects of interest, and the factthat a region is a vertical disruptor may be used as one factor ofmultiple factors that indicate that a region is part of an object ofinterest and/or that the object may be a foreground object. If thesuspected IE extends beyond one or more adjacent grid boxes, or extendsalong the X-axis for 3 or more grid boxes, which for example may be 0.1inch to ⅛th inch (when the image is viewed in the size that the imagewill be printed or presented) and/or fills 1 or more grid boxes morethan 20% and/or extends beyond the boundaries of one or more grid boxes,the intrusion is evaluated as a Vertical Disruptor. Vertical Disruptorsare irregular edges, so all Vertical Disruptors are irregular edges, butnot all irregular edges are Vertical Disruptors. In an embodiment, instep 612 a, the irregular edges that are not Vertical Disruptors aremapped. In measuring a VD, the size of the boxes should be chosen sothat the area of the Vertical Disruptor arrived at by using the numberof boxes that the width and height of the Vertical Disruptor fit iswithin 40% of the area of the vertical disruptor when using the actualheight and width to compute the area of the vertical disruptor (as anapproximation of the actual area of the vertical disruptor). The area ofthe intrusion may be computed in other ways (such as by counting thenumber of pixels used to represent the intrusion divided by the numberof pixels in the region that intrusion intrudes into). A stitched imagemay be used to remove regions known to contain one or more VerticalDisruptor. In step 1, the dominant contiguity is identified on athresholded or edged image (stitched or original). In step 2, the gridis boxes (or pixels occupied by the intrusion are identified and countedand/or identified. In step 3, intrusion areas are classified asnon-regular (irregular) or classified or as Vertical Disruptorsdepending on the size of the intrusion.

In step 614 a, the edge irregularities and optionally the differences inedge irregularities are stored.

In step 616 a, the average position and/or contour of the irregularedges are calculated. In at least one embodiment, machine system 101 canbe configured to calculate the average irregular edges. The averageposition and/or contour of the irregular edges may be computed byaveraging the differences in the edge irregularities (e.g., includingone value of no difference corresponding to the baseline value itself),and then adding the average values of the position to the baselinevalues (of the location and contour of the irregular edges) of thecontiguities.

In step 618 a, vertical disruptors in the contiguity and/or contiguitylines are mapped. In step 618 a, a map of vertical disruptors iscomputed as a baseline computation of the position and other parameters(e.g., the contrast or degree of disruption) of the vertical disruptor.In at least one embodiment, machine system 101 may be configured to mapthe vertical disruptors. The vertical disruptors may be objects orelements identified in the image that extend into a vertical plane froma horizontal line, e.g., from a contiguity. Vertical disruptors arehorizontal features that disrupt contiguity lines and/or contiguities.The map may be based on the regions (quadrants) of the region map, and amap for each region may be computed. In at least one embodiment, machinesystem 101 can be configured to map the vertical disruptors. Differencesbetween the vertical disruptors of different degrees ofstitching/peeling and/or thresholding may also be computed.

In step 620 a, the vertical disruptors and optionally the differences inthe positions of the vertical disruptors are stored.

In step 622 a, an average vertical disruptor may be calculated byaveraging the differences in the vertical disruptor (e.g., including onevalue of no difference corresponding to the baseline value itself) andthen adding the average of the differences to the baseline values of thevertical disruptor, and/or the spatial separation between multiple VDsstored. In at least one embodiment, machine system 101 can be configuredto calculate the average width span, height and/or density(co-localization) of the vertical disruptors.

In step 624 a, a contiguity continuity value (CV) is computed (e.g.,based on steps 616 a and 622 a). In at least one embodiment, machinesystem 101 can be configured to assign the contiguity continuity value,which is the value assigned to the contiguity and represents the degreeto which there are disruptions in the contiguity across the X-axis,e.g., where the X-axis is the horizontal plain of the image. Forexample, the contiguity continuity value can have a value within a rangeof −1.0 to 1.0. The contiguity continuity value may be assignedaccording to the values obtained for the vertical disruptors andirregular edges. For example, where the contiguity extends across theimage within a range of 75 to 100 percent, a contiguity value range of 1may be assigned. Where the contiguity line extends across the imagewidth within a range of 50 to 75 percent, a value of 0 may be assigned.Where contiguity extends across the image within a range of 0 and 50percent, or the contiguity is zero, a value of −1 may be assigned. Inalternative embodiments other values and methods of computing thecontiguity continuity may be used. For example, the percentage of thewidth of the image that the contiguity extends (or the percentage of thewidth of the image that the contiguity extends minus 50%) may be used asthe contiguity continuity value (so that the contiguity continuity valueis a continuous variable).

The method of FIG. 6B is part of the method of FIG. 6A. Step 602 ofFIGS. 6A and 6B may be the same step.

In step 604 b, the position and shape (and optionally other parameters)of the contiguity disruptions (CD) are mapped to establish a baseline ofthe shape, dimensions, and/or position of the disruptions. Contiguitydisruptions are breaks or partial breaks into a contiguity. For example,a region in which the width of the contiguity is less than the adjacentregions (e.g., by more than 10% or 15%) may be considered a contiguitydisruption (in other embodiments other criteria and/or percentages maybe used for determining a contiguity disruption). Note that theterminology used here the length of contiguity extends generally alongthe horizontal axis or at an acute angle with the horizontal axis of theimage, and the width of the contiguity extends along the vertical axisof the image or at an acute angle to the vertical axis of the image. Inat least one embodiment, machine system 101 can be configured to map thecontiguity disruptions. The contiguity disruptions are mapped to enablemachine system 101 to locate the contiguity disruptions in the image,e.g. where there are objects or portions of the image that disrupt thecontiguity in the image. The map may be based on the regions (quadrants)of the region map, and map for each region may be computed. In at leastone embodiment, machine system 101 can be configured to map thecontiguity disruptions, which may also include vertical disruptions incontiguities or contiguity lines. Optionally, differences in one or morecontiguity's linearity and continuity may also be computed and comparedusing different degrees of stitching/peeling and/or thresholding.

In step 606 b, the contiguity disruptors and optionally the differencesin contiguity disruptions are stored.

In step 608 b, an average contiguity disruption is computed, byaveraging the differences in the contiguity disruption (e.g., includingone value of no difference corresponding to the baseline value itself)and then adding the average of the differences to the baseline values ofthe contiguity disruption. In at least one embodiment, machine system101 can be configured to calculate the average contiguity disruption.

In step 610 b, angular changes (AC) in the contiguity and/or contiguitylines are mapped, to establish baseline values. In at least oneembodiment, machine system 101 can be configured to map angular changeof the contiguity line. The angular change (AC) can be the angle atwhich the contiguity in the image relative to an X-axis (a horizontalaxis), e.g., horizontal plain of the image. The map may be based on theregions (quadrants) of the region map, and map for each region may becomputed. Optionally, difference between angular changes in contiguitiesof different degrees of stitching/peeling and/or thresholding may alsobe computed. In step 812, the angular changes and optionally thedifferences in angular changes are stored.

In step 614 b, an average angular change (AC) is calculated, byaveraging the differences in the angular change (e.g., including onevalue of no difference corresponding to the baseline value itself) andthen adding the average of the differences to the baseline values of theangular change. In at least one embodiment, machine system 101 may beconfigured to calculate the average angular change. The average angularchange can be the average angular change of the dominant contiguity,another designated contiguity or all contiguities in the image.

In step 616 b, a contiguity linearity value is computed, which may bebased on steps 808 and 814. In at least one embodiment, machine system101 can be configured to assign the contiguity linearity value, which isthe value assigned to the contiguity for a deviation of the X-axis,e.g., horizontal plain of the image. For example, in an embodiment, thecontiguity linearity value can have a value within a range of −1.0 to1.0 and is derived from the average contiguity changes (Step 608 b) andangular changes (Step 616 b) using measurement boxes, which may becomputed in steps 406 (FIG. 4) and/or step 606 (FIG. 6) of Ser. No.16/427,305 (Docket # CY-6-7). The measurement boxes (or regions of othershapes) may be boxes formed by the grid. The contiguity disruptor andangular change may be computed for each region (in steps 804 and 810)and then the values of each region for the contiguity disruptor andangular change may be averaged over the entire images in steps 808 and816, and then the two averages (the contiguity disruptor and angularchange) may be used to compute the contiguity linearity in step 818.Although in FIGS. 6-8 average values are computed by computing abaseline value and then averaging the differences of subsequentmeasurements taken at different degrees of stitching and/orthresholding, in other embodiments, the average values may be computedin other ways, such as by remeasuring the edge irregularities, thevertical disruptors, the continuity contiguity, and/or the continuitylinearity, and averaging the entire measurement.

In an embodiment, each of the steps of method 600 is a distinct step. Inanother embodiment, although depicted as distinct steps in FIG. 6, step610-690 may not be distinct steps. In other embodiments, method 600 maynot have all of the above steps and/or may have other steps in additionto or instead of those listed above. The steps of method 600 may beperformed in another order. Subsets of the steps listed above as part ofmethod 600 may be used to form their own method.

FIG. 7 provides a method 700 for analyzing the content of images.Sub-library images can contain regional content and/orculturally-sensitive content, and/or images designed to meet aparticular cognitive protocol for a subset of users with a particularcognitive profile. Images and image sets can be selected by the systemas part of a protocol and/or treatment schema, and/or to assess theuser's cognitive status; and/or in the case of a registered user imagesets and game boards containing images and image sets can beuser-defined (e.g., personalized), based on a set of rules designed tomatch user interests and preferences to their skill level at a point intime, and through a protocol which defines progressions for users. Useractivities can be stored so as to retrieve their data and/or to allowfor the completion of a saved interactive or for the regimen to progressto the next level according to the user's training and/or therapeuticprotocol.

In step 710, an image is uploaded or scanned by a user or healthcareworker. The image can be an image selected by a user to be part of thatuser's personalized cognitive test, game or interactivity. The user mayfind the image by searching the internet, or the user may choose apersonal image from a photograph, for example, in the user's library. Insome embodiments, the user is a healthcare worker who is preparing acognitive platform configuration to be used for diagnosis, treatment, orresearch. The healthcare worker may choose the image that is appropriatefor a particular patient or group of patients. The image may bedescribed as an image of interest.

In step 720, the image of interest is sized and/or cropped to apredetermined size automatically or by a user. The cropping may removeportions of the image or the portions of the image that are not wanted,or edges of the image that cause the image to be too large forgenerating the composite image, and/or to centralize dominantcontiguities and color blocks in the image or in a portion of an imageor other salient features.

In step 730, each image in the library is tagged with descriptiveelements. The descriptive elements can define the image's action,content and color; each image can also be tagged with an optionaldisplay box which identifies image content. To address non-Englishspeaking users, alternative text labeling in the user's native orpreferred language can be included to maximize the value of theinteractivity for specific cognitive interventions, language remediationand/or training purposes. The image may be tagged with descriptors withhuman input to ensure specificity and accuracy, because image programsmay not provide sufficient accuracy in generating image captions ordescriptions. In some embodiments, once the image of choice is taggedwith descriptors, the image is stored in association with its data inthe library 780.

In step 740 the contiguity analysis is performed. Each image in thelibrary is ranked according to its complexity based on contentcomplexity and contiguity characteristics among other factors. Theimpact of contiguities can be seen using a sunflower field example wherethe field fills the entire frame versus an image where a part of the skyis visible creating a more traditional horizon-type contiguity. This canbe viewed as an interface (of field:sky) with a demarcation in bothcontent and color distributed and identifiable as such based on theuser's personal knowledge and experience base. Complexity in this senseis a function of the content, its spatial relationship and the presenceor absence of contiguity. For example, an image containing a singleyellow flower with a brown center framed in a single color backgroundwould be ranked as less complex then a field of flowers. This is becausethe single flower has a stronger contiguity than the field of flowerswithout a horizon, but the single flower has a weaker contiguitycompared to a field of flowers with a horizon-type interface (providinghierarchical relationships to figure-ground positioning of image contentbased on its contiguity characteristics and in comparing contiguitycharacteristics to other images when combined into composite image sets(see FIGS. 22A-24C).

As part of the contiguity analysis 740, in addition to assignedattributes, each image in the library is analyzed and assigned aestheticand ambiguity values based on a subset of image characteristics; and asecond value, a Compositing Factor related to combining an image withone or more other images, based on a subset of image characteristics,and is derived in part from the aesthetic and ambiguity values. Theambiguity value is also known as the Complexity Rating (CR) and which isbased on contiguity characteristics of at least one dominant contiguityand/or contiguous region which can contribute to the ambiguity and/oraesthetic scoring. The CR value can also be derived for a multiplicityof contiguities contained in a single image for their potential positiveand/or negative impact in a composite image construct in terms of theCompositing Factor; the relative negative or positive contributiondepends on contiguity characteristics, including: content coherence,context, color composition and spatial separation.

In steps 750, 760 and 770, the CR value, the Aesthetic value, and theCompositing Factor (CGCR) are stored in association with otherinformation about the image. In some embodiments, once the image ofchoice is provided with a CR value, the image is stored in associationwith data in the library 780. In some embodiments, once the image ofchoice is provided with an aesthetic value, the image is stored inassociation with data in the library 780.

In step 780, the image of choice and the data associated with the imageof choice are stored in a library. In some embodiments, the library maybe specific to a user, a health-care worker, a disease type, a researchprotocol, or a testing protocol.

The image library can obtain source materials from artists, which caninclude: individual photographers, photography groups, painters,illustrators, and other artists; image uploads from specific user groupsand/or individuals, including research and clinical administrators, andwhich can be integrated into user-specific interactivities and/orlicensed images from other 3rd party vendors including archives andcontent providers to meet specific use cases and/or user requirements.The use of library images is tracked internally for inclusion in any ofthe hands-on, hands-free and/or view only interactivities, includingmanipulatives as well as printed material as environmental enrichments.The tracking also serves to minimize duplication of content and/ornear-content with the use of similar and/or related images and/orassociated Word Lists used for assessments. The component trackingsystem can be used for tracking usage and any remunerations owed to thecontributing artist based on previous arrangements and agreements.

In an embodiment, each of the steps of method 700 is a distinct step. Inanother embodiment, although depicted as distinct steps in FIG. 7, step710-780 may not be distinct steps. In other embodiments, method 700 maynot have all of the above steps and/or may have other steps in additionto or instead of those listed above. The steps of method 700 may beperformed in another order. Subsets of the steps listed above as part ofmethod 700 may be used to form their own method.

FIG. 8 provides an embodiment of a method 800 for building a userprofile (On-boarding). The platform utilizes puzzle-styledinteractivities to engage cognition using images and composited imagescenes which portray real-world content. Users engage with images and/ortheir composites which can be presented as two or three imagecombinations through a selection of interactivities and which can berated according to the interactivity, skill level and image complexity,and metered by the cognitive status and/or capacity of the user indeveloping metrics. Assessments are designed to cooperatively engageprocesses and skills across multiple cognitive domains. The assessmentsare embedded in the interactivities themselves, applying a modified“activity is the assessment” model to capture both intra-activity and/orpost-activity data. Through combinations of the interactivities, theplatform is used to capture a range of data: speed, accuracy measures,reaction time, error type as well as process data inferred by userscreen movements/placements and image part selection patterns.Assessments (see FIG. 18 and the Appendix) of cognition may also bebased on user activities, prior to performing an interactivity (such asthe process of selecting which interactivity the user wants to select orthe process of the user reading the instructions and initiating theinteractivity) and/or activities performed after an interactivity (suchas the process of closing the interactivity, answering questions afterthe interactivity, and/or logging off the platform). Assessments may bebased on multiple activities, designed to target global cognitivefunctioning while at the same time addressing individual cognitivedomain requirements for training and/or remediation purposes.

In step 810 a user is registered. The process of registering may includesetting up a login, including a username and password. The platform mayinclude the integration of image-based passwords which can be integratedinto a pre and/or post-interactivity assessment. Other information maybe included such as information about a user's health, age, medication,and cognitive and physical status. Other information may also include adoctor, clinician, and/or researcher that the user is associated with aswell as contact information. In some embodiments, if the user is ahealthcare worker, information about the hospital, specialty, researchsubject, education, and registrations may be included.

In step 820 a baseline assessment and health survey is performed. Theassessment may include information about the cognitive and physicalhealth of a user, language preferences, and information about diet,sleep and exercise habits, and vision issues. In other embodiments, theinformation may include a simple test or use of the platform toestablish a baseline assessment for the user.

In step 830, the user is assigned a skill level based on the informationreceived in steps 810 and 820. The user (and/or the user's healthcareworker) may review the skill level to identify whether the chosen skilllevel is appropriate. If not, the user or healthcare workers canre-evaluate the information in steps 810 and 820. The user, as part ofstep 830, may be assigned a skill level automatically (based on tests)or by a healthcare provider.

However, in some embodiments, as part of step 830, the user may assignhim or herself a skill level and/or the skill level may be automaticallyassigned based on a formula provided by the cognitive platform.

In one embodiment, as part of step 830, each of the skill levels: Easy,Medium and Hard can include an expandable list of sub-levels andinteractivities at each sub-level within a given skill level accordingto a professional or user-defined protocol. Each skill level does notneed to have the same number of sub-level interactivities. When a subsetof interactive criteria are met at a pre-defined threshold level range—ametric based in part on: time to task completion, correct responses(error rate), time/move, reaction time, combined withpost-activityassessments (testing for language, memory and attention)with Word List Recall (WL-immediate, delayed and extended) and SQ2questions (spatial quantitative, qualitative), Object ID and DimensionalDescriptors, which are designed to evaluate memory, vocabulary, andconcepts, together with attention-focusing requirements for object-cuedObject ID (OID) and Dimensional Descriptors which uses composite images.The user can be progressed and/or regressed to a sub-level within askill level and/or the previous/next skill level. The new level possibleand/or achieved may be assessed for user consistency and userprogression/regression to meet the new requirements and for trackingchanges over time. If consistent, the user can engage with theinteractivities at or within that skill level until the user reaches adifferent threshold, and the interactivities and associated metrics ofinteractivities, number of images, image complexity, sectioning strategyand/or number and size of interactive elements (game pieces) isdelivered to the user. Sectioning strategy refers to the shapes of thestrips into which the image is cut, typically something that isprescribed to the user, but may be manipulated by the user by choosingdifferent difficulty levels. Sectioning Strategy can apply to the shapeof the cut section, as well as the size of the section or part of asection, but primarily refers to the number of slices and width of thesection (same or variable) generated from cutting an entire image 25%cuts=4 sections; 20%=5 sections. A sectioning strategy may include thetypes of sections into which an image is divided into for solving apuzzle. The image may be divided into horizontal or vertical strips andpieced together accordingly. The matching interactivities use wholesections, parts of a section, or can span multiple sections with shapevariability of the presented parts. In a FreePlay type of scenario (orother scenario), where the user selects the features of the“game/interactivity,” size, number, the percentage of the image thatindividual pieces make up, and/or the shape of the pieces may all bevariable that the user may control. Similarly, in a protocol or fixedprogression type of gameplay the variables may be pre-configured andrelated to skill level and image complexity. Similarly for baselineassessments users can be assigned to a skill level based on age, gender,cognitive status using referenced norms, and adjustments made to theskill level for follow-up assessments as dictated by the user'shealth/cognitive status, including changes in vision, and fine and grossmotor control for example.

For the FreePlay mode, the user may be allowed to override suggestedskill levels at defined points in time. For a training mode, anyassessments which are not embedded directly in the user data (forexample, speed, accuracy, reaction time, movement mapping), andspecifically where the user is not required to answer questions afterthe interactive, can facilitate compliance and data gathering. In someembodiments, FindlT-type games can be integrated into the platform toeliminate voice and/or text-based inputs both under FreePlay, and alsoin Clinical Assessment and/or training mode. A FindlT type game is agame in which a user is requested to find something, such as a game inwhich the user is asked to find a face of an individual in an image. Inthe FindITtype game, the user may be given a list of items to find in animage or image set—the image list may include relatively easy to findimage parts (that is salient image parts that have a relatively highsaliency score) and parts of the image that are relatively hard to findand that therefore have a relatively low saliency score, size, and/orwhere there may be competing content (flower with brown center versusflower with yellow center); or, for example in a “reaction time”evaluation to match as many image parts within a specified time. In areaction time evaluation, the user is given a task to accomplish and theuser is timed and scored based on the time taken to complete the task.Reaction time evaluations can include: find the red-roofed house in alandscape image or find the red-roofed house from within a group ofnon-red roofed houses where the user is timed on how long it takes theuser to find, and optionally click on, one or more red-roofed houses) orclick on all brown-centered flowers (and the user is timed on how longit takes the user to find, and optionally click on, all brown-centeredflowers). As an aside, a reaction time evaluation may also be used as apositive or negative Turing test, depending on conditions.

In step 840, the user is assigned to a protocol, which may be a protocolfor providing therapy or evaluating a patient. Step 840 may occur incombination with step 830 or separately. In some embodiments, the useris part of a research protocol, in which case the skill level is notneeded. The user may be assigned a protocol by a healthcare worker

In step 850, the user completes the interactivities that are providedbased on the user's skill level or a protocol in which the user isparticipating (e.g., a healthcare worker or research protocol or study).In each of the interactivities one or more image sections or imagesegments can be used. The sizes can be varied to change the complexitylevel of the task and corresponding to skill levels.

The interactivities may include one or more of any of theinteractivities contained in the platform. Some examples ofinteractivities that may be included in the platform include: MissingPieces, Extrapolation, MatchMe!, MatchlT!, Mutation, Compose, Construct,Object ID, Dimensional Descriptors, Parts of the Whole. Allinteractivities are weighted according to each cognitive domain,ensuring that a battery of selected interactivities reflects andsimultaneously engages multiple cognitive domains.

In one embodiment using single images in the Missing Piecesinteractivity, an image part may be vertically oriented and/orhorizontally-oriented spanning across multiple grid units and/or fill asingle grid unit. The size of the image parts can represent between 1.5%to 50% of an image, for example. In one embodiment using composites inthe Missing Pieces interactivity, an image part may be verticallyoriented and/or horizontally-oriented and can span across multiple imagesections. The size of the image parts can represent between 1.5% to 50%of each image.

In general, the user can move an image part to the reference image andthen over to the grid, with additional back and forth movements betweenthe image part being over the grid and the image part's originalposition (or another position) to effect a process toward arriving at acorrect placement through estimation and/or approximation of thelocation of the image part on the grid, as well as, for the otherinteractivities to check their decisions and/or problem-solving skillswith a given game piece against a reference image or other interactiveresource.

Extrapolation is a variation on Missing Pieces, but instead of using asingle image, uses a composite image set. The user is tasked withextrapolating the position of the image part from the composite andplacing it in its proper location on a grid. The grid can vary in size,the number of pieces to be placed, the number of images and whether thecomposite is a stable or multi-stable type.

MatchMe! is an interactivity in which the user matches isolated sectionsof an image with portions of a reference image; MatchlT! uses compositesand applies the same strategy of matching whole sections and partialsections, including spans. Spans refer to playing pieces which spanacross multiple images sections in a composite. Mutation is aninteractivity in which parts of an image are altered and the user istasked with identifying the altered portions, such as by correcting orremoving the alteration. Compose is an interactivity in which the userassembles isolated parts of an image into a complete image; whereasConstruct tasks the user with constructing a composite image from theisolated sections.

The Jumble-Sort interactivity, the system presents the user with a mixedgrouping of one or more images that can include both vertically andhorizontally sectioned pieces which can be the same width (all 25%, 20%,10% for example) or mixed. The user is tasked to separate not only theimages, but to separate these according to their sectioning strategy. InSpeed Sort, the sorting process, accuracy and number of pieces sorted ismeasured against a fixed time; for example, 15, 20, 30 or 60 seconds,depending on testing conditions and the user's status.

In one embodiment, the user may work with a reference image whichrepresents the original image in its entirety and/or the reference imagemay be a part of a larger reference image, which is extracted and whichmay become the new reference image. Corresponding image parts to beplaced by the user are presented to the user for placement on a grid. Inone embodiment, a portion of the reference image is masked and/orremoved as a visual reference, and the user is tasked with placing imageparts from this masked section on a grid. This same type of imagemanipulation where a part of the reference image is masked can beapplied to other interactivities such as Missing Pieces, MatchMe!,Compose and/or Construct requiring that the user rely on their memoryand attention to detail to complete the interactivity with speed and/oraccuracy.

Interactivities can be used with a Reference Image which can bepresented to the user as a preview—requiring that the user remember whatthey have seen, on-demand/hint, or continuously. Memory and attentiondemands differ for each and are weighted accordingly. All hands-oninteractivities using composites are differentially weighted based onwhether the composite combination includes 2 or 3 images and whether theimage set is stable or multi-stable. In addition, the use of a compositeReference Image musters a virtual, View-Only interaction as users referto the Reference Image and the mind subconsciously identifies parts ofthe whole and tries to resolve the ambiguous nature of the configuredimage set and where the cognitive demands differ between multi-stableand stable image sets.

The interactivities may include one or more of any of theinteractivities provided herein and configured as a battery ofindividual multi-domain interactivities, which may be Mem+ analyzed forprocessing speed type, reaction time assessments (time limitedplacements), accuracy, and figure ground (f-g) dimensional and

The Word List Recall (which may be referred to as Memory Recall) may bea memory recall of 3, 5 or 7 words. For example, at an initialpredetermined time (e.g., T=0′), the user may be given a set of wordsand optionally immediately asked to recall the words. Later, after anintervening distraction activity, at a second predetermined time (e.g.,at T=5′), the user may be asked to recall the words of the same wordlist (the second recall may be referred to as a Delayed Recall).Optionally, the WL Recall may also request more recalls of the word listat an additional predetermined time. In an embodiment all of thepredetermined times are spaced apart by approximately the same timeinterval, during which the user is distracted with otherinteractivities. For example, there may be another Word List Recall atT-15′, an Extended Recall. Alternatively there may be another word listrecall at T=20′. In an embodiment, the time from initially receiving theWord List and the last Word List Recall may be between 10′ and 20′. Inother embodiments other durations of time may be used and/or chosen(e.g., automatically) based on the user's cognitive abilities and/orcognitive issues, as can the number and nature of the word lists,including image-cued word lists with a varying number of image-cuedwords.

In one embodiment of the compose interactivity, the user is presentedwith a sectioned image and/or images and is tasked to construct amatching pattern using a reference image which can be presented for aspecified period of time and/or can be made available throughout theinteractivity session and/or previewed for the user prior to the startof the interactivity, and/or is available to the user on demand. If theinteractivity involves constructing a composite of two or more images,which is referred to as the Construct interactivity, then the user isalso required to sort through the image sections in being able to findthe appropriate sections matching to each of the component images. Theuser's placement pattern and order of placement (end pieces, one entireimage first, then the second) provides insight into process. Behavioralpatterns associated with age can also play a role in user gameplay,patterns and process analysis with regard to risk tasking users,choosing rapid placements at the expense of errors versus risk averseusers, who might choose slower placements to make fewer mistakes- andvariations in between. The user can also be directed to place partsbelonging to only one of the images, an attention task to ignore theflanking and/or distracting pieces. User behavior and performance ontasks can be impacted by frustration, anxiety and stress. User cognitioncan be evaluated under stress conditions by changing the assessmentconditions, the mix of interactivities, the number of speed rounds, forexample.

Images can be sectioned in a pre-determined manner with thecorresponding number and size of image parts/sections provided.Throughout this specification, the phrase “tasked with” refers to theuser being presented with one or more interactivity screens having thetools for performing the task in question and being presented withindications of what task to perform, including practice opportunities.In one embodiment, the user may be tasked to place a special imagesection which is presented separate from another image section and/orsections and highlighted for the user. The user is tasked to place the“special” image parts into the correct locations within a definedtime-period. Depending on the configuration whether the configuration isfor assessment, training, and/or treatment purposes, the user may bepenalized or not penalized on the task, and a type of reward assigned ornot assigned for the correct placement of the image part. The task canbe used for accuracy, reaction time, speed evaluations, and to assessuser responses to changes in piece size, color, shape, sectioningstrategy for advancement assessment and threshold fine-tuning.

In one embodiment of the Construct interactivity (composites), theplatform can use a single sectioning strategy for each of the images ora mixed multiple sectioning strategy for each individual image in acomposite and within an individual image.

In one embodiment, the user is presented with whole sections or withparts of each section (smaller interactive elements) for theinteractivities. The use of smaller interactive elements and/ordifferent sectioning strategies can be used to vary the complexity ofthe interactivity and the attention to detail required, as well as theimage content and color characteristics.

Interactivities involving composite image sets may use a 2- or 3-imageswith any number of sections between 4-100 and/or image partsrepresenting 0.1% or less of an image, depending on the intendedinteractive tasks, level of complexity and purpose of the interactivity.

In one embodiment of the Mutation interactivity, the user is tasked toidentify introduced changes to the image set and/or individual imagesthat result in an error of some type in the image or image setscomposition. Examples of mutations, include, but are not limited to: theduplication of elements that are not duplicated in the original image,the insertion of an unrelated image part, the deletion of an element ofthe image, the inversion of an element of an image, rearrangements ofelements in an image, and/or transpositions of two or more elements ofan image with each other, among others. In one embodiment, thetransposition-type mutation involves a composite where a section or asegment and/or segments/sections belonging to Image #1 is reciprocallytransposed or non-reciprocally inserted into Image #2 or Image #3. Inone embodiment, the inversion type mutation is where an image section orsegment can be vertically flipped Smaller segments can be involved,increasing the challenge level in looking for what is different from thetemplate or what is not correct in a presented construct. An imageand/or image set may contain more than one mutation and can include morethan one type of mutation, and can be provided with a count on thenumber of mutations present, and a countdown as these are found. Theuser may be presented with a series of progressively more challengingMutation interactivities and/or the Mutation interactivity can becombined with another interactivity as part of a therapeutic and/ortraining protocol.

In one embodiment of a MatchMe! interactivity, the user is tasked toidentify one or more matching sections and/or segments of a collectionof images to a reference image which provides the user with an activetemplate that can be copied. Segments in single images can range insize, in both the horizontal and vertical orientation between 1.5%-50%of the total image and/or can consist of a mixed variety of sectionpercentages. In one embodiment, the reference image template is intactand the user looks for a matched section. In one embodiment, thereference image template is missing sections leaving holes in thereference image template, similar to an incomplete puzzle where thepieces are to be placed. The pieces can be the same size or of differentsizes as one another. In one embodiment, using 2- and/or 3-imagecomposites, the MatchIT! interactivity involves a missing segment andthe piece that is to be inserted to that position. The missing or to bematched segment can be a span of multiple sections from more than one ofthe component images. In other words, the missing section or to bematched section that needs to be matched and filled may include two ormore adjacent image sections in a composite image.

The platform can be used for multiple learning styles based on theinteractivities' mix which is used and which allow users to demonstratetheir skills in one or more areas, but which can also highlightdifferences in user ability without penalty for a matched learning styleinteractivity but which is adjusted based on user's capacity andcapabilities.

A unique feature of the platform is the integration of composite imagesets with the inherent ability to interact with the user on a subliminaland/or subconscious level, an effect due in part to the illusion ofdepth which is generated by juxtaposing image sections in both stableand multi-stable image sets, and which is amplified with the dynamicshifting of images in the figure-ground positions which users canperceive with multi-stable image sets. As the user is interacting withthe image sets, a switch in which image appears as background, i.e. inthe ground position can occur. This process engages a second process forthe image in the ground position as the gaps in the confluency of theground image are largely ignored based on Gestalt principles and appearsassembled. This can occur independently of the user's consciousawareness and/or tracking across alternative contiguity lines, and whichmay trigger a perceptual switch. The perceptual switch, while occurringsubconsciously, can also be linked to active engagement of cognitiveprocesses where the user is guided and/or made aware of the switchand/or alternate percepts, and is directed to focus on a particularareas of the image. This kind of a value-added engagement of cognitiveprocesses can be factored into the cognitive benefits offered by theplatform, and which can be evaluated with functional Magnetic ResonanceImaging (fMRI) and/or directed Electroencephalogram (EEG)/Event RelatedPotentials (ERP) signals and/or eye-tracking to provide users with anadjusted baseline. Functional MRI is a type of specialized MRI scan,which measures the hemodynamic response related to neural activity inthe brain or spinal cord of humans or other animals. Higher amounts ofblood flow are seen as indicating activity areas of the brain that havehigher amounts of brain activity. Thus, fMRI measures areas ofactivity—the theory being that the user is lighting up areas of thebrain involved in performing the interactivities. Monitoring the brainwith fMRI or EEG/ERP spikes associated with the user's interaction wouldoccur in a clinical setting. The fMRI or EEG/ERP can also be used todifferentially evaluate multi-domain cognitive engagement especially inpatients/users who have suffered traumatic brain injury, concussion orfollowing stroke where other areas of the brain may compensate for lossof function in one area. A regular user might use a commerciallyavailable EEG headset (like Muse) which is configured for the platform,or use an eye-tracking system which can be used today with an App orGoogle glasses to track user eye movements across and around image/imagesets. However, a home user could also be given access to these types ofclinical measurements.

By supporting and/or improving awareness and thinking skills among userswith cognitive impairments and/or other changes in cognitive status,and/or in building cognitive reserve through attention focusing, memorytriggering and language-building interactivities, the platform has thecapacity to impact ADL (Activities of Daily Living), a practical valueand measure of the transfer of gaming skills and learned skills toproviding practical improvements to the lives of platform users.

In step 860, the data based on the interactivity (e.g., performancedata) is stored in association with the information about theperformance of steps 810, 820, 830, and 840.

In step 870, the user's profile may be updated based on the results ofthe interactivity. Step 870 may include a change in information aboutthe user's diagnosis, cognitive health, physical health, skill level,and/or which protocol the user should be assigned.

In step 880, a progress metric is displayed for the user, giving theuser the results of the interactivity in combination with informationabout the user's skill level, progress, diagnosis, and other profileand/or signature features. The progress metric may be displayed only tothe user and the user's clinician or researcher. Alternatively, theinformation may only be provided to a health care worker who may thendecide how to communicate the information to the user. After theprogress metric is displayed, the session ends in step 890.

In an embodiment, each of the steps of method 800 is a distinct step. Inanother embodiment, although depicted as distinct steps in FIG. 8, step810-890 may not be distinct steps. In other embodiments, method 800 maynot have all of the above steps and/or may have other steps in additionto or instead of those listed above. The steps of method 800 may beperformed in another order. Subsets of the steps listed above as part ofmethod 800 may be used to form their own method.

In some embodiments, the invention uses images and image sets as thebasis for hands-on and/or hands-free and/or view-only challenges andinteractivities; though other parts of the invention can be configuredwith non-visual content for a subset of puzzle-styled interactives(interactivities that are in the form of a puzzle) and/or be combinedwith non-puzzle-type interactivities. Visual inputs can be conveyedthrough a variety of means including, but not limited to, sighting,implants, signal transducing wearables and other devices, and/orbrain-computer interfaces. For visual inputs, the interactivities can beapplied to individual images, to composite image set constructs made upof two or more images, and/or to the individual images (componentimages) comprising the composited constructs.

In one embodiment, the platform may utilize content-rich, real-worldimages. Images can be color, halftone, black/white (b/w), and/ordegraded images/photographs, and/or other types of source images. Someexamples of types of source images are as video-captured still images,artwork, illustration, drawing and/or paintings in various combinations.In one embodiment, individual images can depict single objects as linedrawings, illustrations and/or photographic images, and/or other type ofrepresentations with and without applied graphical filters and masks.Some examples of representations that may be used by the platform mayinclude a degraded image or an image with partially obscured imagecontent. A degraded image or partially obscured image may be used for asubset of interactivities primarily related to one or more of thefollowing: building object and pattern recognition, language skills andattention skills, and executive function among other learning and/orskill-based objectives.

Images with content-rich elements can depict a scene with people,animals and/or inanimate objects in various combinations and/or be setin an urban or rural environment, in a multiplicity of combinations. Theterm “images” can include static images, a combination of images, asequence of images, or moving images. Some examples of moving imagesthat may be included in the term “images” are video or film scenes andclips, as well as static screenshot-type images captured from video orfilm sources. Images can be sourced from pre-screened libraries ofselected images and/or be supplied by the user, according to imagespecifications and security requirements, and the images may be taggedaccording to the platform's requirements with image characteristics(color, content, and contiguities) analyzed as described previously.

Composite images can be generated by serially sectioning and juxtaposingthe image sections from two or more images to portray the illusion ofdepth. The illusion of depth may be a visual illusion effect that isrooted in the figure and ground relationships. The illusion can portraya second characteristic if one or more of the component images containspecific image attributes referred to as contiguities.

Content, color, and context may aid in defining a contiguity or theinterpretation of a contiguity. As such, in an embodiment contiguitiesin an image can provide visual cues which the user can use to trackacross an image when two or more images are combined in a specifiedfashion and where at least one image contains a contiguity or when twoor more images contain contiguities in a composite of two images or in acomposite of three images. A contiguity does not need to span across theentire width of an image and a specific contiguity's characteristics canchange across the width of an image from being pronounced to lessdefined, and vice a versa. The visual effect of the composite and theimpact of the composite on the user differs, depending on each image'scontiguity characteristics with a set of variables, including: color,content, context and the image's overall complexity, and which can beused to personalize image sets, and by default, the user's interactivegameboards/interactive surface and experience. Users with higher degreesof cognitive abilities may be able to piece together an image in theirmind with fewer visual cues, such as contiguities and/or contiguitiesthat are less noticeable, as indicated by the numerical valuesassociated with the contiguity's characteristics. As a result,interactivities may be chosen based on the characteristics of the imageand the contiguities in the images to test for higher and lower degreesof cognitive ability.

Images can be contained in an image library (PICSSL) and can be taggedwith attributes. Attributes can include include text and/or audio labelsof image content and elements, a category label, and assessmentattributes, for use with Word List Recall using the image's embeddedvisual cues, and associated SQ2 (Spatial, Quantitative and Qualitative)questions, Object ID and Dimensional Descriptors. Each image may beanalyzed based on its contiguity characteristics to derive aesthetic andambiguity values which are used to define the complexity of the imageand its potential complexity contributions in a composited image set.

The image library may be restricted to include a subset of images whichmay be presented to select user groups, depending on skills and learningrequirements. For example, in an embodiment, the use of nature-themedimages in the platform leverages prevailing knowledge about interactionswith nature and the effect of the interactions with nature to improvewell-being based on Attention Restoration Theory and Stress ReductionTheory. The system can be configured to deliver defined images based ontheir content and/or a subset of image content to meet therapeuticand/or training requirements, such as to train or test the end-user'smemory and/or attention. The images and/or subset of images may bechosen needs of place and time. In one embodiment, the platform may beconfigured to use images with people's faces or with portions ofpeople's faces, such as a profile. The images with people's faces orwith portions of people's faces may be included to support facialrecognition and/or other biometrics-styled interactivities for trainingusers with deficiencies in related to being able to recognize faces ascan occur in people following stroke and with Autism Spectrum Disorder.

The library may include a searchable index and categorical groupings tofacilitate use of images and/or searching for/selecting images.Contiguity characteristics and/or relationships of contiguities toestablishing figure-ground relationships in and between images not onlyprovide visual landmarks, which can be used as search parameters,contiguity characteristics and/or relationships of contiguities toestablishing figure-ground relationships can also be used for generalimage analysis purposes in computer vision applications, and which applyprinciples of visual perception and attention.

In some embodiments, the platform may be utilized to deliver a subset ofinteractivities for entertainment purposes and/or as part of a marketingand promotions strategy using visual content to support customerengagement, acquisition, and retention. Content may be user-suppliedand/or content supplied by a third party, which may be used to provideusers with incentives, rewards, discounts and other promotions relatedto the content and or the promotional objective. The incentives mayinclude travel and tourism, entertainment venues and film, and/or otherentities with promotion objectives which can utilize an interactiveand/or engagement tool to engage current and prospective customersand/or users.

In some embodiments, the platform may be used to effect a team-buildingand/or to effect a change in management strategy within a corporation, auser's health, and/or a business. The platform may be used to effect achange in an activity and/or to effect compliance with a rule, such as achange in maintaining a drug and/or treatment protocol and/or adoptionof a new software solution in support of a deployment. The use of theplatform in these types of situations can be viewed as a combination ofentertainment and learning facilitation, according to the platform'scognitive engagement capacity.

FIGS. 9-12 provide embodiments of methods of using the interactivecognitive platform by a user.

FIG. 9 provides a multi-session protocol for a returning user. Bothprofessional users (e.g., health care workers) and end-users (e.g.,patients) may access the platform's multiple interactive modes,including: FreePlay, Challenge and Mem+ mode. The Freeplay mode allowsthe user to choose what game the user would like to play and to play anygiven game for as long or as short of a time period as desired with userdefined selections for skill level and images. The challenge modeincludes interactivities that challenge the skill level of the user withpreset progressions. The Mem+ modes includes interactivities thatinvolve the use of memory. Based on instructions given to the user bythe professional, and/or as the user directs themselves, a subset ofinteractivities may be performed for a specified period of timeaccording to a prescribed frequency and/or as directed by the systemwith recommendations on frequency and skill level, and/or as part of anintervention or maintenance regimen to support cognition. The user canmanually engage the assessment mode for a specific interactivity or setof interactivities, and add those measures to their user profile.

FIG. 9 shows a multi-session protocol for a returning user 900. In step902, a returning user is provided with a login interface, via which theuser may login (see also 810 in FIG. 8 for login information that may berequired).

In step 904, the user performs a session survey. A session survey ispresented to the user. The session survey may include questions askingthe user whether the user found the previous interactivities and/or thecognitive protocol useful. Questions may be presented to the use, viathe platform, to identify specific information that allows the cognitiveplatform to be better personalized to the user. There may be otherquestions that request other information to identify whether theprotocol is helping the user (e.g., via diagnosis or treatment of adisorder). Alternatively or additionally, the survey may ask whether theprotocol has been entertaining and/or engaging to the user to furthersupport compliance. For example, questions may include, but are notlimited to, how long the user sleeps, sleep patterns, diet, dietaryhabits, current medications, medication changes, stress levels, etc.These are variables that can impact user data and may explain why scoresare significantly different from a baseline score, that the changes arenot progressive or permanent, but conditional. There may be questionsasking for information about the user's health, dietary, and sleeppatterns and other things that may affect the user's performance for agiven session and may erroneously imply changes in cognitive status butwhich are situational. However, in an embodiment situational changeswould be tracked to facilitate identification of actual changes incognitive status which are associated with long-term changes in sleep,diet, exercise for the user and which may be applied to other users indeveloping predictive analytics. In some embodiments, the survey caninclude questions enquiring whether the clinician found theinteractivity useful.

In some embodiments, the session survey identifies what type of learnera user is in combination with previous progress metrics. For example, aninteractivity where a user is unable to perform a certain task withaccuracy or within a predetermined time threshold, may be used as anindication of the user's cognitive status. The user's cognitive statusmay be further assessed using alternative protocols within the platformoptionally together with third party assessments of the user. Forexample, visual-spatial tasking can be highly developed in some usersthat are deficient in other areas as a matter of history (despite, forexample, a statistical correlation between those other areas in whichthe user is deficient and visual tasking, indicating that it is unusualfor someone with this user's deficiencies to be highly developed invisual-spatial skills), and the high visual spatial skills is in thecase of this user is therefore not an indication of changes in cognitivestatus over time or due to a condition (even though a temporary furtherexacerbation of their inabilities may occur. As another example, a userwho has historically been an auditory learner and not a visual learnerwould find certain tasks challenging as a matter of course, and eventhough most people that perform poorly on certain auditory test alsotypically perform poorly on certain visual tests, the poor performanceof this user on those visual test and the good performance of this useron those auditory tests is not a result of a change in their currentcondition or cognitive status. The converse may also be true in that astrong visual-spatial learner may have auditory learning deficiencies.Similarly there may be users having an array of mixed types learningstyles along a continuous spectrum.

By leveraging a knowledge of the user's learning style, an assessment ofchanges in cognitive status is informed by the user's general abilitiesas a factor in choosing and evaluating the value and validity of certaintypes of assessment indices. The variability in learning styles isviewed as a factor in personalizing gameboards and interactivities tobetter meet user needs, rather than as a deficiency to be remediedand/or negatively contributing to a user's evaluation. The dynamicaspect of the platform allows for the integration of multiple learningstyles in order to foster cognitive gains whether for remediation,training and/or skills development. The platform does, however, place apremium on visual input, hand-eye coordination and reasoning—criticaltasks needed for everyday living. As such, the platform may not besuitable and/or effective for people who have little or no interest inpuzzle-type interactivities.

In step 906, the user's profile is updated based on the informationreceived via the survey.

In step 907, the user's previous protocol data is recalled. This mayoccur to enable the protocol to choose a skill level, to identify whichprotocol and interactivities the user should do next (perhaps based on aclinical protocol the user is involved in), or based on the user'spreferences.

In step 908, the user's interactivity data is recalled. Optionally, thesurvey may be skipped and the user may be immediately presented with theinteractivities. Along with recalling the user's interactivities dataand presenting the interactivities pages to the user, theinteractivities may be personalized and a new interactivities page (withnew interactivities) may be presented to the user based on the user'sprevious personalized interactivity.

In step 910, the system conducts a Word Language (WL) Recall assessmentat time 0. In the pre-session survey, the user is also tasked with aword list recall assessment administered at T=0′. A significant part ofthe assessment protocol is that the assessment protocol may be builtinto the platform—the assessment may be an embedded assessment that isembedded in the interactivity process, which may be implemented as theuser engages in the interactivities, as described below. The Word Listsare derived in part from objects and elements contained in the imagesand/or image sets which are used as the basis of the interactivities. Assuch, the words in the Word List which the user is tasked to rememberover time are image-cued and embedded in the interactivities. During theinteractivities, the user is physically manipulating, or in the case ofa view-only option, the user is mentally manipulating, the image parts,and interacting with a subset of the objects and elements in their WordList (WL) both on a conscious and subconscious level. The Object IDMemory (OIDm) assessment has similarities to Word List Recall assessmentin that the words are image-based and image cued, only with IODm theuser derives the word list from an image. With OIDm, they are given 15seconds to come up with 5-7 descriptive words about the image. After 3-5minutes of other interactivities, they are tasked with remembering asmany words as they previously listed.

Back to Word List Recall, each image may be tagged with descriptorswhich identify objects and elements contained in each image and fromwhich images or parts of images are associated with a list of words,creating an image-associated Word List that may be derived for the userand/or with one or more user sessions. A Word List Recall assessment maycontains three (3) or more words. The number of words to be recalled isalso metered to the user skill level and cognitive status. For example,Image #1 can contain representations of a bird, a branch, sunrise, day,a silhouette, a hawk, no leaves, a tree, and golden. The wordsassociated with an image can be represented in the image as visualelements or derived from the image inferred such as the “day” tag. Thenumber of words used in the assessment is based on and/or derived froman image and/or image set is approximately forty (40) percent; such thatfor a 3-Word WL, at least one word is based on, and/or derived from theimage and/or image sets. Similarly, in a 5-Word WL, at least 3image-cued words would be integrated into the assessment. Optionally,the words derived from the image may be derived automatically, based onan automated image recognition program, which may make use ofcontiguities, for example to identify parts of the image.

In an embodiment of the Word List Recall assessment, the user is taskedto recall the word list (WL) at multiple time points. The first timepoint is T=0′. Similar to traditional Recall assessments, the user,after hearing and/or reading the list of words aloud, is tasked torepeat the words back into optionally a microphone of system 100, twice,for example A Delayed Recall assessment is deployed at or about T=5′.During the intervening time, between T=0′ and T=5′ (other time periodsof other lengths of time may be used instead), the user is tasked withperforming a set of interactivities using the images with tagged wordsrelated to and/or conveyed by the image or image sets, depending on theinteractives the user has been tasked with in a session. Traditionalcognitive assessments which integrate a Word List Recall and DelayedRecall protocol generally use only T=0′ and T=5′ time points results tocalculate a cognitive metric. The cueing used allows the recallassessments to use visual cues of tangible real-world objects readilyavailable in the user's immediate environment with words such as chair,desk, table, window—familiar and within the user's visual field, whichcan test the associations the user makes between words and images. Theplatform uses visual cues as well but the visual cues are embedded inthe image sets with which the user is tasked to interact with,generating a “do” operation, enabling an enhanced potential for userlearning, memory retrieval, attention focusing, and skills developmentthrough the interactivities and their use and/or manipulations of theimage and/or image sets as well as the reference image to support auser's solving stratagem.

The platform provides the user with directed interactions with theassociated cues with the goal of fostering a stronger connection andmemory recovery and memory building opportunities through an activelearning approach and potential associative scaffolding.

In one embodiment, the platform uses an extended time point forconducting a recall assessment which is timed to occur at T=+10′ fromthe start of the user's interactions with the image-basedinteractivities and/or within that time frame up to approximately T=15′in one embodiment and/or to the completion of an active interactivity.In one embodiment, parts of the recall assessment can extend to longertime-periods within a given session, or may extend beyond a givensession to another designated time point within a treatment protocol orbeyond the treatment protocol depending on the requirements of thetreatment and/or training requirements. The number of correct andincorrect absolute responses, such as the recall of the precise wordand/or words is assigned a score. As the recall assessment assignspoints, users may receive partial points for errors in word list recallassessments, if the words the users recall are categorical, i.e., theword, “table” might be called furniture by the user or might be called,desk or dining room table with an elaboration on the to-be-recalled wordor words.

In one embodiment, users can be tasked to place object labels onassociated objects. Placing object labels on associated objects can beused to reinforce learning for those with cognitive and/or languageimpairments. Text labels may be configured as part of a multi-languagepack to make the platform user-friendly to non-native English speakersand/or to people who have developed linguistic challenges associatedwith cognitive changes. In one embodiment an auto-sequence may run wherethe system places object labels on image objects to support recovery andlearning for users with limited fine and/or motor control. Theinteraction in this case is View-Only but offers mock-ups of theinteractivities for viewing in addition to viewing engagement of theimage sets themselves.

A Tangible User Interface (TUI) device and/or prop can also beprogrammed to present a word label. The user would then place the devicewith a label proximal to the object as part of the interactivity and/orassessment. In one embodiment, the labels can be on-demand activated orautomatically displayed by the system in order to assist a subset ofusers with specific cognitive issues, including linguistic challengesand where tagging of image elements can help support cognitive health.

In step 912, Session 2 begins with interactivity part I. In step 914,Session 2 proceeds to interactivity part II. In step 916, Session 2proceeds with interactivity part III.

In step 918, a dynamic skill level adaptation assessment is performedbased on Session 2, in particular based on the interactivities part IIand III.

Parts I, II and III is a shorthand used to describe interactivity sets,where parts I, II and III would be interactivities set 1,interactivities set 2 and interactivities set 3. Each set may becomprised of a collection of interactivities using the same image setsthroughout. Since there is a 5 minute time gap between WL Recall T=0′and T=5′ and similarly between T=5′ to T=15′, the time can be filledwith interactivities. For example, a healthy young person might completea standard interactivities set 1 in a minute, making the time intervalrequired for performing a WL Recall (immediately then delayed) in needof filler interactivities to have a delta of 5′ to get to T=0′ and T=5′recall time points. Optionally, the user may automatically be givenother interactivities to fill the remaining time. In other embodiments,a time interval that is different than 5 minutes, and where thedifferent time intervals may have different durations.

In step 919 alternatively a word list (WL) Recall assessment isperformed at time=5 minutes after Session 2 interactivities Part I (step912).

In step 920, a WL Recall assessment is performed at time=15 minutesafter Session 2 interactivities Part III (step 916). In addition to aT=+5′, T=+10′ and/or a T=+15′ recall as part of the extended Word ListRecall, a modified Word List Recall assessment can use compiled wordlists from multiple sessions and associated images, and where the useris assessed at another session and/or at the midpoint and/or conclusionof a therapeutic protocol.

The Word List Recall assessment, together with other assessments, can beconducted in sessions in the presence of a health care worker, in adigital, remote or physical mode. Assessments can also be conducted in aself-directed manner by the user themselves using audio recordings andanalysis of the user's responses for Word List Recall assessments,and/or for SQ2 questions which can be a verbally transmitted and/orusing a device including direct input into a device with transmission ofuser data, and/or through a secondary device, such as a scanner wherethe user's inputs are recorded on paper and then scanned for analysis bythe platform and/or health care worker.

Similarly, for a facilitated assessment the user is tasked to recall theWord List at specified times and to verbally state and/or manuallyrecord their responses. The latter can be accomplished with a compiledlist of words from the word list as well as non-word list words. Theuser is then tasked to identify from the mixture of words, only thosewords contained in the word list. The audio recording of the user'sverbal responses can be used as a biometric tool to indicate changesover time in various vocal and/or voice related metrics.

In step 922, the user data is stored. Storing the data occurs in eitheralternative, after steps 920, 919, and/or 916. The data can be used infuture sessions to identify an appropriate skill level and/orpersonalized interactivity for the user.

In step 924, the user's profile is updated with the information providedin step 922.

In step 926, the user's progress metric is displayed.

In step 928, the session ends, although if the user and/or cliniciandesires, a third session can be started in step 930 or 932 with Session3 interactivities within the same sitting.

In an embodiment, each of the steps of method 900 is a distinct step. Inanother embodiment, although depicted as distinct steps in FIG. 9, step902-932 may not be distinct steps. In other embodiments, method 900 maynot have all of the above steps and/or may have other steps in additionto or instead of those listed above. The steps of method 900 may beperformed in another order. Subsets of the steps listed above as part ofmethod 900 may be used to form their own method.

FIG. 10 is an example of a flowchart showing three options for how aregistered user may interact with an interactive cognitive platformresulting in the display of a user progress metric 1000. The user maychoose FreePlay Mode, Challenge Mode, Protocol Mode, and User-definedmode. In addition to choosing the mode, the method provides many stepsthat allow the user to personalize the GUI, the game, the interactivity,and/or to include the Mem+ protocol. The personalization process mayinclude interactivity sequence and skill level progressions as well ascustomization of the User Interface to allow the user to adjust andmanipulate the left-right, top-bottom configuration of interfaceelements. The personalization process may include spatial manipulationsof the reference image, template/grids, work area containing interactiveelements, resizing of elements, zoom capabilities, image library hide,timer toggle, among other platform features.

The platform presents the user and health care worker with a Mem+ mode,which can deliver multiple therapeutic and training interventions basedon the user's requirements and dynamic cognitive status. The Mem+ modecan also be implemented by a health care worker to address user-specificissues as part of a chronic and/or transient condition to help supportbrain health and as an assessment for baselining a user's cognitivestatus to track changes over time, the introduction of new medications,drug safety during drug development, cognitive norms followinganesthesia administration, and/or to evaluate the effectiveness of anintervention. Mem+ mode delivers a series of interactivities in adirected fashion, and/or as part of a progression. The Mem+ mode may bedivided between Easy, Medium, and Hard levels which may be entered atany of the sessions automatically and/or according to a health careworker's directions (in other embodiments there may be fewer or moregradations of the level of difficulty of a test or battery of tests).

In one embodiment, each level has a minimum of number of sessions (e.g.,12 sessions per level) and may use the same image set for apredetermined number of sessions (e.g., for groupings of 2 sessions)over a predetermined time-period (e.g., on a weekly basis), as anexample Within each skill level, the interactivities may range from easyto difficult along a continuum or vary only with the image setspresented to the user which are used in grouped sessions (for examplethe grouped sessions may be weekly sessions—sessions grouped by theweek). In general, the therapeutic protocol can use switch-capable imagesets (multi-stable), non-switchable image sets (stable) and/or a mix ofswitchable and non-switchable image sets, as well as, the componentimages, according to the therapeutic or training protocol.

Mem+ can have multiple modes, including an Attention Focusing mode whichmay allow the user to develop and/or advance their skills using a subsetof the interactivities of the Mem+ interactivities. In the attentionfocusing mode, the user is presented with a series of interactivitiesusing image sets that utilize a variable sectioning strategy which canbe portrayed as a progression of changes but where the collection ofinteractivities or the individual interactivity is still multi-domain innature to greater and lesser degrees across the different domains. Theplatform may also deliver a subset of images, which may also containdistractor and/or attractor elements. The image content's objects may berepresented as part of an image or may be represented as the imageitself in its entirety. For example in the case of a flower, the flowermay be viewed as either an attractor or distractor depending on theprotocol and the second and/or third images in a composite, and the usermay then be tasked with an appropriate measure of attention using Mem+and SQ2 questions related to the flower image and/or one of the otherimages in the composite.

In step 1010, a user is registered (see also step 810 in FIG. 8). Asummary of how the user may use the platform is described here. The userregisters to use the resources or is registered by a facilitator(professional, therapist and/or caregiver). If the user is engaged withthe platform in a self-directed manner, the user can access one or moreof the following in a session: FreePlay, Challenge, or Protocols (inother embodiments there may be other options. Each new user is taskedwith completing a baseline set of interactivities, given at one of thethree skill levels, depending on their assessed estimated and/orprojected abilities to obtain baseline measures. A re-evaluation may beconducted periodically and the initial data may also be compared toother point-in-time data for FreePlay and Challenge users across ages,gender, conditions or other parameters for comparative purposes. Usersengaged in Protocols options are assessed within the protocol, relativeto their baseline measures and to other normed data sets.

In step 1015, a user decides to use FreePlay Mode. In FreePlay mode, theuser may resume a saved interactivity or begin a new one. The user isresponsible for choosing the images, interactivities and skill level.The user progress may be measured according to best times to completeand which may be posted and compared to other users scores using thesame scenario (images, skill-level and interactivities) in a modifiedtype of competitive play against other users and also themselves toimprove their personal scores.

Alternatively, in step 1020, the user may choose to proceed in ChallengeMode. In the Challenge mode, users are also provided with a set batteryof interactivities, where completion of the tasks progresses the userthrough levels of increasing difficulty and/or complexity with respectto image content and number of tasks required to complete the level.Completion statistics are available for each user at each level andsub-level, and which can be made available to peers using the site basedon user privacy settings. In addition to the professional collaborativespace, a challenge space can be included in the platform whichencourages use by users through game design and competitive play betweenusers who have developed their own user-defined configurations and wantto share their configuration. This social space can include stats, chatsand potentially live tournaments similar to multi-player game site wheretime to completion is the metric of success. In addition, the cross-overopportunities can include testing of professional gameboards withvolunteer end-users; and/or, the development and sharing of a widerrange of interactive tools suited to different user groups withdifferent health conditions through these types of collaborations.

Alternatively, in step 1025, the user may choose to proceed in ProtocolMode. In Protocol Mode, protocols may be developed by clinicians forusers and/or suitable to groups of users across ages, gender, languageand motor capabilities, and/or conditions. The users may be users thatare not responding to standard therapies. Users may be individuals whohave been selected by professionals to participate in one or more healthcare programs. In some embodiments, as part of the protocol mode 1025,if a user attempts to enter the protocol mode, and if it is determinedthat step 1025 proceeds to step 1065. In step 1065, the session ends,because the user is not assigned to a protocol mode yet or because thetiming of the protocol does not allow a user to participate at thistime. Protocol users may also be users in research and/or clinicalstudies. Protocol users may also be represented by general users whohave an interest in the health care program, i.e., self-recruited intothe program studies to beta test software and/or as participants in thestudies directly. Protocol users may also access a separate developmentarea for non-professionals (or professionals) as developers fordeveloping new protocols for new therapies and/or test. Theinteractivities can also be made available to Professional ProtocolDevelopers for use (and/or further subsequent study). In step 1030, theUser may choose to proceed in User-Defined Mode. Interactive protocolsdeveloped by non-professionals or professional may be placed into a testbed area and which can be shared with other general users (optionallythe protocols developed may be protocols for tested and proventreatments, but which may need to be tested for software bugs oroptionally may be part of a research program). In some embodiments, aspart of the protocol mode 1025, the session ends in step 1065 becausethe user is not assigned a protocol mode yet or because the timing ofthe protocol does not allow a user to participate at this time.

User-defined Mode may include any choices the user made previously topersonalize one or more interactivities, the GUI, the images, etc.

In step 1035, alternatively to step 1030, a user is assigned to aprotocol by a health care worker.

In step 1040, the user chose Challenge Mode (step 1020) or the User wasassigned to a protocol (e.g., in step 1035) and the system automaticallyproceeds with a system-defined mode. System-defined mode may be based onprotocols provided by health care workers or researchers for this user.Alternatively System-defined mode may take into account all of theprevious information about the user and, via an algorithm, decide whichmode would be best for the user. In some embodiments, after the user isassigned a protocol, a Mem+ assessment occurs (step 1055).

In step 1045, images, skill level and interactivities are chosen by auser via User-defined Mode.

In step 1050, skill progression interactivities are provided by thesystem based on either the images, skill level, etc. chosen by the userin step 1045 as part of the User-defined mode, or chosen by the userbased on the system-defined mode implemented in step 1035.

In step 1055, a Mem+ assessment occurs by the system. Assessments arebased on a set of interactivities which the user is tasked to complete.Time to completion of the task contributes to building the metric as canthe following: how the task is completed, the number of correct orincorrect placements, repeat errors in mis-placements, reaction time,and rated according to the overall skill level and skill leveladjustments during the interactive, stable or multi-stable image sets aswell as the overall complexity rating of the multi-domain interactive orbattery of multi-domain interactivities. The user's interactivitypattern is factored with multiple variables, including: time of day,sleep patterns, medication changes, stress/daily impacts, and otherchanges in health status which can be assessed through pre-sessionsurveys.

In step 1060, the user data is stored and in step 1065, the sessionends.

In step 1070, the user profile is updated after the session ends, and,if desired by the user or the healthcare worker, in step 1075, the userprogress/metric is displayed.

In an embodiment, each of the steps of method 1000 is a distinct step.In another embodiment, although depicted as distinct steps in FIG. 10,step 1010-1075 may not be distinct steps. In other embodiments, method1000 may not have all of the above steps and/or may have other steps inaddition to or instead of those listed above. The steps of method 1000may be performed in another order. Subsets of the steps listed above aspart of method 1000 may be used to form their own method.

FIG. 11 is an example of protocol options 1100 for a user starting withthe selection of one or more image from a graphical user interface. Thusthe option 1100 can be whether the user or the system selects the imagesthat are then used for interactivities for the cognitive platform. SeeFIGS. 21-25 for examples of images that have been sectioned in variousways.

In one alternative, the user selects one or more images in step 1110from a library of images. Alternatively, the user may upload one or moreimages.

In a second alternative, the system selects one or more images in step1115 from a library or may upload one or more images.

In step 1120, the user selects the sectioning strategy for the image orimages that the user selected in step 1115.

In step 1125, the system selects the sectioning strategy for the imageor images that the user selected in step 1110.

In step 1130, whether the images are selected by the user or the system,the system generates composite images sets based on the images and thesectioning strategies chosen. At this point the system may immediatelydeliver Mem+ interactivities to the user (step 1155) or the User mayselect Mem+ protocols (step 1135).

In step 1130, whether the images are selected by the user or the system,the system generates composite images sets based on the images and thesectioning strategies chosen. At this point the system may immediatelydeliver Mem+ interactivities to the user (step 1155) or the User mayselect Mem+ protocols (step 1135).

In step 1140, alternatively, the system may select the Mem+ protocolbased on the composite image set (step 1130).

In step 1145, the system delivers a word list recall activity to theuser at time 0. The word list recall and WL recall methods are discussedin detail in FIG. 9 (see step 910). However, the embodiment of step 1145includes a timed aspect. Optional visible timers may be used, as noted,above where each turn must be completed within a specified time frame,and in an embodiment, in some interactivities users are provided with aspecial piece that may need to be placed within a specified time toscore the number of full points or to meet a specified threshold forerror, time and/or reaction time. The reward and/or rewards attendantawards the rewards, when the piece is correctly placed by a user incompliance with an incentive program. In one embodiment, such a programis designed to encourage user compliance with a “you win” strategy topromote adherence to a protocol, to introduce new platform featuresand/or to include a special points earned to support and/or encourageuser progress or completion of a task associated with an interactiveprotocol of any type.

The use of rewards in interactivities may include audio rewards, pointsin a game type setting, to indicate progress and/or regression whenpoints are removed. The rewards may be incentivized tangible rewardsoffered through third parties such as coupons, discounts, tickets orother premiums, and/or intangible rewards such as a completed image setposted in a user gallery, and/or high scores/best times posted on aleader board depending on the configuration which can be designed forcompetitive game play for team building and/or for remediation andtreatment with incentives offered for compliance with a therapeuticregimen. The rewards approach can be applied to the entire userinteractive experience or to parts such as with the “special”interactive element (game piece) which has a reward above and beyondwhat may be included in generalized interactivity.

The use of a score or other composited value may be provided to theprofessional and/or user to assist them in measuring the effectivenessof the user's efforts and to identify areas of improvement and/or inneed of improvement. The metric is factored with other “point in time”measures as well as the assessments delivered to the user as part of thebaseline measurements and subsequent formative and summative assessmentsduring a given protocol. The assessment design may be intra-activity,post-activity, or with 3′ party assignments to domains+(multi-domain)comprehensive global (markers which demonstrate global cognitiveengagement skills and processes, along with domain-referencedskills—local).

In step 1155, based on the word list (WL) recall, the system deliversMem+ interactivities. In one embodiment, the Word List Recall is notincluded in an assessment battery and an alternative such as ObjectID-Memory and/or Dimensional Descriptors can be used. In one embodiment,a language-based memory assessment may not be included though othermemory assessments are included in interactivity battery to captureworking, short- and long-term memory functions. The Mem+ interactivitiescan be any of those discussed herein. In some embodiments, the Mem+interactivities may be chosen by the user, based on user preferences,and/or chosen by a health care professional.

In step 1160, a delayed recall assessment is delivered to the user attime 5 minutes. Meaning that the user has 5 minutes to complete the wordlist recall test.

In step 1165, the user provides the word list (WL) responses.

In step 1170, the user responses are scored. In step 1175, the user datais updated. In step 1195, the user baseline and/or progress metric isgenerated.

In step 1180, the system delivers Mem+ interactivities based on the WLresponses in step 1165. In step 1185, a delayed recall assessment ismade at time 15 minutes, but can be at other times in other embodiments.In step 1190, the user provides the WL responses. Based on the delayedrecall assessment interactivity, the user's responses are stored, andwhich can include actual voiceprint recording data, and the user'sresponses are updated, and a progress metric generated (see steps1170-1195).

In an embodiment, each of the steps of method 1100 is a distinct step.In another embodiment, although depicted as distinct steps in FIG. 11,step 1110-1195 may not be distinct steps. In other embodiments, method1100 may not have all of the above steps and/or may have other steps inaddition to or instead of those listed above. The steps of method 1100may be performed in another order. Subsets of the steps listed above aspart of method 1100 may be used to form their own method.

FIG. 12 is an example of a method of making a user interactive workspace1200. The workspace may be produced in one of several ways. See FIG. 21for an example of a graphical user interface or workspace. The user maybe tasked to work through a series of interactivities using theindividual images, and/or through a series of interactivities related tocomposites comprised of 2-3 component images where the individualcomponent images are serially sectioned and juxtaposed to generate aninterspersed pattern of non-adjacent component image sections.Interactivities may include hands-on as well, hand-free activities, suchas view-only interactivities, where a hands-on interaction with physicaland/or digital manipulatives is not necessary as a part of the processof working with the platform. Such interactions occur in theuser/viewer's mind (user input may be received via a microphone and/orcamera or in the case where a user is interacting with the platform ontheir own, there may not be any user input). As such, the view-onlyinteraction of method 1200 is also termed as an interactivity becausethe view-only interaction requires the user's engagement, whetheractively or passively conveyed to the user and whether the user isconsciously or subconsciously engaged in the interaction. Assessments inView-only mode, however, require the use of additional biometrics typetools such as eye tracking and EEG, though speed and accuracy measureswould not necessarily be available to users or for analysis. The lack ofspeed and accuracy data does not diminish the value of the platform, butdoes limit the availability of a subset of data for analyses andreporting.

In step 1210, the User selects FreePlay. FreePlay is discussed in FIG.10 (step 1015). Alternatively, in step 1215, the user selects a Mem+protocol.

Alternatively in step 1120, the system selects a Mem+ protocol based onthe user information stored and provided when the user logs in.

In step 1225, in all cases (steps 1210, 1215, or 1220), a determinationis made as to how many images are selected by the user. The image orimages may be combined into 2-3 image composites where the individualcomponent images are serially sectioned and juxtaposed to generate aninterspersed pattern of non-adjacent component image sections.

Composite images can be made up of 2 or more component images which havebeen serially sectioned and the image sections from one imageinterspersed with another one or two images. Component images may besectioned into two or more sections across their entire width and thesections juxtaposed next to sections from a second, and/or third image,such that sections from any one component image is not placedimmediately adjacent to one another. Sectioned parts may besubdivided/sectioned further, and the user may be presented, forexample, with half-height pieces, quarter-height pieces and smallersegments or pieces which span across multiple sectioned images. The gapbetween otherwise adjacent sections in a component image may be between1%-50% of the image's total width, depending on specifications to effectthe re-assembly of the hyphenated image segments to occur, despite thegapped appearance of the construct. The gap may be filled by a secondand/or a third image, cut according to specifications or by white orother solid color spaces which can be viewed as a background substrategenerated by placement of one of the images with a defined spatial gapbetween the image sections. In one embodiment, a solid white backgroundcan serve as a virtual or physical substrate for both online and offlinemanipulatives.

As noted previously, the serially sectioning and juxtaposition ofmultiple image sections in an alternating fashion generates the illusionof depth based on figure-ground relationship. The presence of one ormore contiguities in a composite image can confer an additional aspectof the visual illusion in terms of the stability of the image whichoccupies the ground and/or background position. In one embodiment, theconfiguration may be referred to as stable when only one of the imagesin a composite contains at least one contiguity. The image with thecontiguity occupies the ground position, and in the stableconfiguration, the second image in a 2-image composite; or the secondand third images in a 3-image composite will occupy the figure(foreground) position. The image or images in the figure position canappear as columnar pop-outs supporting the portrayal of the illusion ofdepth. In a stable configuration the same image always occupies thebackground position. A multi-stable configuration one where at least twoof the images in a 3-image composite or both images in a 2-imagecomposite can occupy the background (ground) position; the switchcapability as such is high for the image set. Both stable andmulti-stable constructions can be generated using either the platform'sdevice-based and/or offline components, and/or a hybrid version.

In both stable and multi-stable embodiments, the image in the groundposition can be viewed by the viewer as being intact, confluent, despitethe spatial hyphenations between sections and where these spatial gapsare largely ignored. This gap filling (perceptual completion) can occurwhen the intervening spaces are filled with one or two content-richimages and/or when the gap space is filled with a solid color, such aswhite (empty space). The dynamic re-assembly of the hyphenated imagesegments of the image occupying the ground position can occur based onthe presence of visual, context, and knowledge-based cues as part of theuser/viewer's experience base and predictive inferences, continuity,together with gap-filling (perceptual completion) capacity of theinformation conveyed by and through the contiguities present in theimage.

For example, a green-blue colored interface extending across the entirewidth of the image can potentially be identified by the user as afield/sky interface based in part on the color of different regions andspatial characteristics, and the user's knowledge of field and sky. Theregularity and continuity of the interface can be anticipated by theuser and the intervening disruptive and/or distractor image sections arelargely ignored as the viewer tracks the next image section containing agreen-blue interface. Interference can be established with the choice ofintervening images and overlapping contiguities between the componentimages including the image set. Together the Gestalt principles offigure-ground, closure, continuity and gap filling (perceptualcompletion) can be used to understand the scientific basis of the visualillusion and its applications in the platform for assessment,diagnostics, remediation and training purposes. Continuity refers to themind's tendency to complete a continuous region.

In describing figure and ground relationships, the terms of recessiveand dominant can be used, respectively. Dominant may be used to refer tothe image which assumes the ground position in a fixed and/or dynamicfashion in a stable configuration, or images in a multi-stablecondition, respectively. In the stable condition, the figure position isthen occupied by the other image in a 2-image composite or 2 images in a3-image composite, if neither of these images contains a dominantcontiguity relative to the image occupying the ground position.

For a multi-stable composite, the presence of at least one contiguity inat least 2 of the images in a 2- or 3-image composite, the compositeimage will generate a multi-stable image, and/or that each of the threeimages in a 3-image composite each has at least one contiguity will alsogenerate a multi-stable image.

The term multi-stable refers to the ability of more than image, or imagesection or part to assume the ground position (with a concomitant flipor switching/shifting of the previous ground occupant to a figureposition). A multi-stable, figure and ground relationship in an imageset is where the images can be perceived in more than one way, i.e. theperception of the image set changes/switches. This flip or switch canoccur spontaneously and the different forms are referred to as percepts.In these types of embodiments, the image which occupies the groundposition dynamically shifts between the image and/or images in thefigure position at a given point in time and is perceived in analternate fashion by the viewer. Both stable and multi-stable image setsshare the illusion of depth, but differ in their switch/shift capacityas described previously and in the following sections.

Not all percepts are equally stable and dominance is relative to thecomposite's composition. For example, if the component images in astable 3-image composite are extracted and reassigned to a 2-imagecomposite, a previously figure-bound image in a 3-image composite canassume the ground position, because of a relative state of contiguitydominance. In other words, a weak contiguity can be in the groundposition relative to a composite with a second component image withweaker contiguity characteristics, but be relegated to the figureposition in a stable composite if the weak contiguity is dominated by animage with a contiguity with stronger characteristics. In part onereason that a weak contiguity may occupy the ground position relative toa composite with a second component image with weaker contiguitycharacteristics is due to the presence of a minor contiguity whosecontiguity characteristics while present were otherwise perceptuallymasked in the 3-image composite or a 2-image composite, but which can beexpressed in certain combinations of the derived 2-image compositeand/or in combination with other images.

As such, in one embodiment, an image with a weak contiguity can becombined with one or more images which do not contain any contiguities,making the image with the weak contiguity the dominant image and whenthe sections are combined, the image with the weak contiguity can assumethe ground position. The hierarchy in which the image with the highestcontiguity assumes the ground position can be driven in part by thecontiguity's characteristics and user's/viewer's input and/or biasand/or preferences. The multi-stable capacity is nonetheless conferredon an image based on the individual image's absolute contiguitycharacteristics and are metered by the combination of the image withother images in terms of the expression of the contiguity. FIGS. 25A-Cuses the same set of three images in multiple combinations which can bedescribed as 2:3, 1:2 and 1:3 to demonstrate hierarchical dependencies.In this example, Image 1 in FIGS. 25B and 25C is dominant to Image 2 andImage 3. Both Images 2 and 3 would appear to portray similar figureposition tendencies, but when Images 2 and 3 are combined in FIG. 25A asfigure-ground hierarchy places Image 2 in the ground position relativeto Image 3.

In both stable and multi-stable images, the image which occupies theground position, with or without a perceptual switch, can be perceivedand conveyed to the user as a coherent image, intact despite the spatialseparation between the image sections, and their disruption withinterspersed content-rich image sections and/or blank spaces if thesecond (or third) image(s) are solid in color, and which can be viewedas distractor or attractor elements, depending on their use. Thisgap-filling capacity despite the hyphenation is consistent with theGestalt principle of completion, which is that the mind tends tocomplete an image.

The integration of multi-stable images into the cognitive platformallows for dynamic cognitive engagement of the user with the compositedimage sets, whether in a conscious and/or unconscious mode as perceivedby the user/viewer. This engagement is facilitated by selecting andusing image sets of differing complexity for select training andtreatment modalities, together with the interactivities mix. Contiguitycharacteristics form the basis of developing a complexity rating forimages based on their ground position capacity and/or switch capacitydepending on the image configuration together with image content, colorand context variables. The prospective image combinations can be definedaccording to a set of rules where a composited image scene can becategorized as stable and/or multi-stable, and with a determination inthe stable condition which image will assume the ground position, withassignments to varying complexity levels for the variousinteractivities.

In multi-stable image sets, the switch frequency can vary between usersand as a function of cognitive status related to: age as a factor and/orneuropsychological conditions, such as schizophrenia and autism.

Switching events and, as such, switch frequency for multi-stable imageshave traditionally relied primarily on user-identified switch eventswhich are signaled by a click of a mouse or other type of device toindicate conscious awareness of a switch event. In general, the imagesused for measuring altered switch rates are binary ambiguous images, inthat the switch occurs between two alternate perceptual states(percepts) within the same image. Examples of these types of imagesinclude the Necker Cube and Rubin Wine Glass-Face illusion. Themulti-stable image sets used in the platform involve a switch betweendifferent images, guided in part by the user tracking (e.g., moving theuser's eyes) across a given contiguity or towards salient image parts.As such, interactive measures and/or an analysis of switch rates amongdifferent population groups can be improved and used as a diagnostictool using both user-identified switches combined with objectivemeasures such as eye tracking analysis to detect a shift in the user'sgaze or eye focus from the spatial location of a contiguity in Image #1to a contiguity in Image #2 and/or Image #3. Optionally, throughout thisspecification, any time an eye is tracked, the eye may be trackedautomatically via a camera in system 100, and analyzed by the processorsystem of system 100 or 200. Switch events such as eye tracking can alsobe monitored using EEG tools in part, because of the integration ofreal-world images into these dynamic image sets and therecognition/discovery process which can occur when the ground imagebecomes confluent coincident with and/or part of a switch event. Thepotential for identifying evoked/event response potentials together witheye tracking data, as well more sophisticated biometric and analyticaltools, can be used to improve these measurements and assess theirpotential value as part of a diagnostic profile of cognitive functionand status.

In one embodiment depicted in FIGS. 23A-F the multi-stable nature of animage set can be shifted to a stable configuration by selectivelyremoving one or more contributing contiguities which otherwisepositively enables the image set's switch capacity. The shifting betweenswitch and non-switch and back to switch capable image sets can beincorporated as part of a training protocol (or a testing or therapeuticprotocol). FIGS. 23A and 23F are identical but are placed separately toshow the multi-stable to stable progression and differences between the“fully-switch capable” multi-stable image set to the stable“fully-switch incapable” stable image set. Shifting between switch andnon-switch and back to switch may be used to impact one or morecognitive domains, including the translation of response times fromchanges in processing speed parts attributable to switch speed forattention focusing.

If it is determined in step 1225 that a single image interactivity isdesirable, the method proceeds to step 1230, and a single imagealgorithms loaded. In step 1235, system offers the user a choice ofimages to select. After receiving a selection, the system proceeds tostep 1240. In step 1240 a single image interactivity is produced basedon the single image selected.

If in step 1225 it is determined that the user would like to selectmultiple images, in step 1245 the user is offered a multiple image toselect from and is allowed to select multiple images. In step 1245,algorithms for creating multiple-image interactivities are retrieved. Instep 1250, Image #1 is selected, in step 1255, Image #2 is selected, instep 1260, Image #3 is selected. In some embodiments, more than threeimages are selected. In some embodiments, two images are selected. Insome embodiments, three images are selected. In one embodiment, incomposited image scenes at least one of the images (Image #1) maycontain real-world content portrayed as a photograph, graphic, paintedor as a constructed image including a tangible prop or other type ofphysical and/or digital manipulative, while at least one other image(Images #2, #3 and/or #4) can contain content and be presented in aformat similar to Image #1 or can consist of a solid color or mix ofsolid colors including: white, black or gray tones of varyingpercentages or other type of illustration. The sectioning strategy maybe uniform or variable for one or more of the images. The juxtapositionstrategy may be sequential, non-sequential, may include partial or fullmasking (skipping) of one or more image sections, and/or use a solidcolor image giving the appearance of unfilled gaps between imagesections. Sectioning strategies may be uniform for each of the componentimages between 1% and 50% and/or portray a mixed sectioning strategy,depending on the construct, where each image may follow an independentsectioning strategy and may itself portray a mixed sectioning strategy.The variation in sectioning can contribute to the designated skill levelfor one or more of the platform's interactivities. In general, thethinner and/or smaller the image sections, the more challenging theinteractivities and the user's cognitive demand requirements to focustheir attention on the contiguities and/or component elements of animage for assessments, including embedded assessments for memory and/orattention based in part on the amount of available information which canbe used for cueing and pattern analysis to identify parts of the whole.In one embodiment, the impact of the sectioning strategy, 10%, 12.5%,20%, 25% and 50% for example is adjusted with a weighted factor,reflecting the availability of image content cues and image detailswithin the image sections for facilitating the user's analyses of theimage set's content and the complexity level to solve theinteractivities. In one embodiment, a 20% sectioning strategy may lenditself to ease of use as compared to both a 25% sectioning strategy or a10% sectioning strategy because of the ease of access and/or hyphenationof content in more refined sectioning strategies within a sectioningstrategy range, a “sweet spot” so to speak.

In one embodiment, a progressive reduction in the sectioning width,and/or an increase in the number of sections conveyed to the user forpart of the image, can be used as part of training protocol to conveyattention focusing skills development through an increase or decrease inthe sectioning used and the subsequent interactivities used for one ormore of the images in a composite image set, and/or individual imageswith associated interactivities.

In step 1270, a composite image set interactivity is produced based onthe selected images.

Based on these images, the system creates a user interactive workspace.In some embodiments, “user-interactive” means that at least oneuser-preference was incorporated into the workspace. In step 1275, thesystem generates a reference image. The user can be provided with areference image. The reference image can be presented in several modes:continuous, intermittent, preview, limited and/or on-demand displaymode. The reference image can serve as the source of the visual cueswhere the user can match to and/or work with associated manipulativeelements related to the reference image towards working on andcompleting a task. The non-continuous display of a reference image foran interactivity and/or set of interactivities increases the cognitiverequirement for memory and attention. As described for sectioningstrategy, factors are applied to adjust the weighted values for eachcognitive domain's representation/contribution to the multi-domaincharacter for each interactivity. In one embodiment, the user isprovided with a reference image by which to model their tasks for asubset of the platform's interactivities but with variable engagement ofmultiple cognitive domains as compared to intermittent, preview oron-demand use of reference images.

In one embodiment, where the reference image is presented as a compositeimage set, the user's referencing of the image to perform associatedtasks can provide for additional interactivity based on the platform andthe composite image sets' view-only capacity for interactivity based oncognitive engagement—a factor which is differentially weighted into themulti-domain character of the interactivity and its contribution with abattery of interactivities (see the Appendix). The additional image setinteractions based on the user's referencing of the image set can beviewed as a value-add and which is delivered via the cognitiveplatform's presentation of a reference image to the user and the use ofthe reference image by the user.

In an embodiment, reference images can be presented to users for aspecified period of time and/or can be available throughout theinteractivity session and/or previewed for the user prior to the startof the interactivity, and/or is available to the user on-demand,depending on the requirements of the treatment and/or training protocol.

In step 1280, the system selects and implements a sectioning strategy,in step 1285, the system selects or generates an interactive workspaceusing the sectioning strategy and the images chosen, which may stillneed to be populated with an image and images sections. In step 1287,the system selects or generates a template grid. In step 1290, thesystem presents the reference image. In step 1295, the system presentsthe image sections. In step 1297, the user interactive workspace isdeveloped using the image or images selected.

In an embodiment, each of the steps of method 1200 is a distinct step.In another embodiment, although depicted as distinct steps in FIG. 12,step 1210-1297 may not be distinct steps. In other embodiments, method1200 may not have all of the above steps and/or may have other steps inaddition to or instead of those listed above. The steps of method 1200may be performed in another order. Subsets of the steps listed above aspart of method 1200 may be used to form their own method.

FIGS. 13 and 14 provide embedded assessment tools which are part of theinteractive cognitive platform. A key component of the platform is theembedded aspect of the assessments across multiple cognitive domains,which is integrated with the interactivities and which extends theassessment capabilities to include other metrics beyond intra-activityspeed and accuracy measures and sub-measures. The user can be tasked towork through a series of interactivities using the individual images,and/or these can be combined into 2-3 image composites. Interactivitiescan include hands-on as well, hand-free and in particular view-onlyinteractions, where hands-on interaction can occur with physical and/ordigital manipulatives but which is not necessary as a part of theprocess of working with the cognitive platform as physical reassembly iscomplemented by virtual assembly of the hyphenated image parts which canoccur in the user/viewer's mind. As such, this view-only interaction isalso termed as an interactivity because the view-only interactionrequires the user's engagement, whether it is conveyed to the user,actively or passively.

The integration of both dynamic and stable aspects of the image sets,together with the interactivities multi-cognitive domain engagement, andby default assessment capabilities, together with, but not limited toplatform personalization through image selection and interactivityfeatures, skill level scalabilty, demographic-neutrality, multimodal(View only, Hands-On) use including: self-directed, group and/orfacilitated use with the use of a hidden timer and/or untimedinteractivity based assessments, and other features make the invention aversatile multi-purpose platform suitable for use across a range of usercapabilities and in multiple user environments.

FIG. 13 is an example of a method of a user interacting with a cognitiveplatform to generate a metric and/or update a user skill level 1300.

In step 1305, an interactivity(0) is selected by a health care worker oruser. Interactivity(0) refers to an interactivity which has not yet beenpersonalized to a user.

In step 1315, a user interactive workspace is provided for theinteractivity(0). In step 1320, system waits to receive input from theuser, such as click on a start button. The user starts theinteractivity(0) by first clicking the start button when he or she isready (step 1320) and then in step 1325, system waits to receive aplacement of an image section on a grid (step 1325) where grids are usedin a Compose, Construct, Missing Pieces and/or Extrapolationinteractivity, for example.

In step 1330, if the system determines that the user incorrectly placesone or more sections, an auto-alert element indicates the incorrectplacement to the user. In an embodiment, in the all-edge design, allplacements are possible because of the elimination of specific fittedshape restrictions. In some embodiments, in the digital version of theplatform, the user can be alerted to a misplacement with visual andauditory alerts, and/or kinesthetic and or vibratory indicators ofcorrect, incorrect placements, and/or can be used to provide proximityhints to the user. In some embodiments, as the user places an elementincorrectly, the misplaced element can be automatically returned to the“active” interactivity space or gameboard area, and can be marked ortagged as having been tested and/or tried by the user. In someembodiments, a visual signal (e.g., a red bar) is placed above usedpieces which have been incorrectly placed by the user. However, othergraphical or sensory methods can be used to indicate incorrectplacement. In an embodiment, the user is given the chance to correct theplacement until the placement is correct (in another embodiment the usermay be given a finite number of chances to correctly place a piece, suchas between one and 30 chances to correctly place the section on thegrid). In some embodiments, the user is given the option to change theskill level if the placement is too difficult (e.g., if the user triesmore than 5 times to correctly place the section on the grid and doesnot accomplish correct placement). In some embodiments, theinteractivity will go on to the next step if the user cannot correct theplacement after a present number of tires, such as one and 10 tries.

The alert systems may be configured to allow the user to receive dynamicfeedback to correct near-completed and/or actual misplacements; or thealert may be dispensed with to allow the user to complete the placementof all game pieces without feedback and then be scored at the completionof the task. Depending on the protocol, the user can be given anopportunity to correct misplacements for the purposes of achieving abetter score, and/or for learning purposes about missed visual cues andcognitive skills development, including second choice or second bestchoice options. A secondary digital interface such as a phone, tabletcomputer, or other type of smart device can also be used to scan orcapture an image of the user's completed interactive task and resultsconveyed to the platform's assessment module for scoring purposes.

In step 1335, the user is presented with an opportunity to correct theincorrect placement. In step 1340, the misplacement error is recorded.In some embodiments, the number of times required for the user tocorrect the incorrect placement is also recorded. Tracking of usererrors and categorization of error types, including repeated attempts ofplacements in the same location provides insight in user strategy anddecision-making (random versus targeted), attention, memory and othercognitive considerations.

In optional step 1345, the misplaced section is returned to theworkspace (in an alternative embodiment, the misplaced section is leftwhere placed, optionally with a visual indication that the placement isincorrect), and in step 1350, the user time, data & placements arescored. A time may be recorded after the user clicks the start button(step 1320) to record time information, after the user places an imagesection correctly (step 1335), after the user completes theinteractivity, and/or after a misplacement error (step 1340).

In step 1355, a determination is made that the user has completedinteractivity(0), which may occur when the image has been properlyconstructed or other interactivity task completed, the system indicatesthe user has completed the task, and/or after the user has spent morethan a predetermined about of time with the interactivity. In step 1360,the embedded assessment tool is deployed—the interactivity is assessed.The embedded assessment tool may compute and/or recompute other testother metrics, such as those in step 1365 and the WL recall tests, basedon input from the current interactivity. Thus, in step 1365, theMem+/SQ2 is integrated with the user's responses to the interactivity.Word List Recall assessments and SQ2 questions are image-basedevaluations integrated into and with the interactivities, allowing for amore sensitive and accurate assessment of the user using the platformwhich can be metered to address a user's changing and evolving statusand/or requirements.

The platform's embedded assessment tools therefore include an evaluationof number of correct/incorrect responses, time taken to complete aninteractive task, pattern of errors together with the recall/delayedrecall and extended delayed recall, and SQ2 responses to provide ausable metric to help assess a user's cognitive capacity relative tobaseline measures and other measurements taken as points in time and/orat defined interval as part of a clinical and/or research protocol,and/or other comparative measures, including a Big Data analysis of userdata obtained from a sample pool and compared across user data, physicaland physiological variables including: age, sex/gender, diagnosis,stress levels, EEG, education, professional positions and/or potentiallyother contributing variables.

These data contribute to the user profile in building both point in timeand changes over time metrics based on user data, as well ascontributing to large cohort analytics (Big Data), including theintegration of third party data towards informing the platform'sevolution, an approach which uses a continuous improvement model forproduct research and development to meet current and future user needs,and for developing predictive analytic benchmarks for a broad range ofuses, including: diagnostics and change monitoring.

In step 1370, the user data is updated based on the integration and, instep 1375, the user progress metric is generated. In step 1380, based onthe interactivity and the embedded assessment, the user skill level isupdated.

In an embodiment, each of the steps of method 1300 is a distinct step.In another embodiment, although depicted as distinct steps in FIG. 13,step 1305-1380 may not be distinct steps. In other embodiments, method1300 may not have all of the above steps and/or may have other steps inaddition to or instead of those listed above. The steps of method 1300may be performed in another order. Subsets of the steps listed above aspart of method 1300 may be used to form their own method.

FIG. 14 is a second example of a method for interacting with a cognitiveplatform by a user 1400 (see also FIG. 13). FIG. 14 providespossibilities for what happens after a user selects an interactivity.

In step 1405, the user selects an interactivity(0).

In step 1410, the user time, data and placements are scored. At thispoint, based on the scoring in step 1410, the user may either completethe interactivity (step 1430) or a change level threshold may betriggered (step 1415) based on the previous progress or results by theuser and/or the progression of the user's cognitive disorder. In someembodiments, the change may be due to the fact that, as a user practicesthe types of interactivities provided, the user may get better at themand may require a higher skill level.

In step 1420, after the change level is triggered, the skill level isdynamically adjusted by the system with an interactive add-on.

In step 1425, the user completes the new interactivity(n)

In step 1430, the user time, data and placements in the interactivity(n)are scored.

In step 1435, if the change level threshold is not triggered, the usercompletes interactivity(0), not interactivity(n).

In step 1440, the embedded assessment tool is deployed after completionof either interactivity (0 or n), after either step 1430 or after 1425.

In step 1445, the Mem+/SQ2 user responses are integrated with theassessment of the interactivity. In an embodiment, Mem+/SQ2 is used asan example of a post-activity assessment (beta category assessment) asdescribed in conjunction with FIGS. 25 and 26.

In step 1450, the user data is updated, the skill level is then updatedin step 1455, and/or the user progress is generated in step 1460.

In an embodiment, each of the steps of method 1400 is a distinct step.In another embodiment, although depicted as distinct steps in FIG. 14,step 1405-1460 may not be distinct steps. In other embodiments, method1400 may not have all of the above steps and/or may have other steps inaddition to or instead of those listed above. The steps of method 1400may be performed in another order. Subsets of the steps listed above aspart of method 1400 may be used to form their own method.

With reference to FIGS. 1-14, in some embodiments, two or more of theplatform's components can be combined. For example in a Stress module, asuggested sequence of interactivities may be provided to the user inresponse to an assessment and/or the user may elect to use the platformin a self-directed manner accessing a user-defined module which allowsthe user to select the image sets, the interactivities, the skill leveland the optional use of a timer. The user's interactivity statistics areanalyzed, recorded and stored, and can be presented to the user to helpinform platform-assisted recommendations and/or the user's owndecisions. The platform can be integrated with additional devices,including tablets, phones and other touch-mediated devices and/oradd-ons and/or equipment which allow for the monitoring of physiologicalmetrics including but not limited to: multi-channel EEG, single channelEEG, heart rate, respiratory rate, blood pressure, galvanic skinresponse, pupil dilation, temperature and other spatial, temporal and/orfrequency brain states as assessment tools.

FIGS. 15-18 provide methods for use of the platform by a professional tocreate a specific cognitive diagnostic or test. In the platform, theterm “professional user” can refer to a clinician, a researcher, ahealthcare worker, a professional game maker or gamer. The platform isdesigned to allow for dynamic configurations and the assembly of theelements to generate personalized game boards, and interactivitybatteries based on user preferences and/or health care worker-basedinputs for therapy and/or diagnostic and/or assessment or otherprofessional purposes (“professional”), as well as configurationsgenerated by system AI logic in evaluating a user's preferences and/orcognitive requirements.

FIG. 15 shows a method 1500 for allowing professional users to builddiagnostic, assessment, treatment, and training tools. Professionalcommunities would be able to use a menu of choices and branched optionsto establish interactivity parameters towards building their ownprotocol configurations (Protocol configuration builder). Configurationscan be stored under a Mem+ label towards building a dynamic library ofdiagnostic, assessment, treatment and training tools to supportcognitive well-being and skills training FIG. 15 is an example of acollaborative method in which professional users (e.g., health careworkers) may analyze data from users based on skill levels 1500. Themethod in FIG. 15 may be used by a health care worker for treatment ordiagnosis of a disease, or for understanding a diagnostic tool,producing an effective treatment, producing an effective diagnostictool, producing an effective treatment tool, and/or for research intounderstanding brain, neuronal and neurocognitive processes for example.In some embodiments, the method can be used in combination with othertreatments (to analyze their effectiveness) or with other diagnostictools. FIG. 15 shows collaboration between four professional users, butin other embodiments, collaboration can take place between any number ofprofessional users from 2 to 100s or 1000s or more. Professional usersmay include, but are not limited to, health care workers, analysts,professional game makers, and/or gamers. Professional users typically donot include patients or those that interact with interactivities fortraining, therapy, and/or testing themselves, but the collaborativespace can provision for interactions between professional users andenrolled users in a trial or study where researchers can virtuallyinteract with users and/or study participants.

In steps 1505-1520, Professional users A-D create Mem+ A-D. Mem+interactivities are discussed in detail in FIG. 10. However, alsoincluded in the Mem+ interactivities are interactivities that can bedone in view only mode (VO) and/or hands free mode. These areparticularly useful for users that have patients with diseases wherepatients no longer have the use of their extremities, cannot speak,and/or have difficulty with speaking or physical movements includingtemporary situations such as following the administration of generalanesthesia where recovery of cognitive functions can be temporarilyslowed. Under these circumstances, one way to diagnose is to observepatterns of what part of the brain is active during an interactivity.For tracking progress, the part of the brain that is active can betracked over time and/or with treatment. The part of the brain that isactive can be compared to that of a normal individual (one who does nothave the cognitive disease) for diagnosis and treatment tracking. Insome embodiments, parts of the patient's brain may be compensating fordamaged parts of the brain and/or the patient's may also be trained aspart of doing the interactivities.

View only may be referred to as Virtual/View-only (VVO) mode. For anyhands-on interactivities or if the user has limited or no fine/grossmotor control, the user does not have access to eye-control technologyor the person has limited or no ability to communicate verbally. VVOallows the user to use other methods for approaching the interactivitiesfor cognitive benefit to them. For example, when looking at the imagesets—stable or multi-stable—the mind is engaged in resolving theambiguities, in discerning figure and ground relationships, inre-assembling the hyphenated image parts into a confluent image. Thistype of engagement can be trapped, tracked, and imaged usingeye-tracking, fMRI, EEG/ERP and other physio biometrics to identifycognition in the user. EEG and ERP are methods of measuring brainactivity. Electroencephalogram (EEG) is a test used to evaluate theelectrical activity in the brain. Brain cells communicate with eachother through electrical impulses. An EEG can be used to help detectpotential problems associated with this activity. An EEG tracks andrecords brain wave patterns. Event Related Potentials (ERP) are measuredin EEG. For example, the health care worker says “flower” and the eyetracking shows eye movement to the flower region of the image set. Ifthe health care worker then says “yellow flower”, there is specific eyemovement to a yellow flower. In some embodiments, to refine (temporallyand spatially) these kinds of signals, EEG/ERP signals (or fMRI) can bematched to image recognition. Further, the system can discern the switchbetween the component images in an image composite, for example, donewith the user clicking a button and saying switch. The process can berefined by applying technology to demonstrate and characterize theswitching phenomenon, which can be done because the platform integratesthe use of both stable and multi-stable images, and transitional imagesets where a multi-stable image can be turned into a stable image and inthe reverse sequence as well. This can be done because there arehierarchical relationships between contiguities and in imagecombinations (1:2:3 versus 1:2 versus 2:3 versus 1:3) image drivendifferences which are processed differently as shown in thefigure-ground positioning hierarchies in FIG. 25. It is no longer justrandom parts that user perceives, but rather parts of the whole, and/orthe whole itself. Because of this, these properties can be linked tointeractivities and assessments. For example—the user has a stable imageset and is asked to describe what they see. That process is differentwhen using a multi-stable image set where the user must concentrate(prevent attention shifting and ignore the flanking content) to describeeach component image's content individually.

The platform's hands-free mode is to be distinguished from the view-onlymode which is also hands-free and involves engagement of the user. Thehands-free mode refers to the use of an alternative user adaptive typeof input devices or other assistive technologies such as eye-control,mouse cursor control, voice-activated controls, and/or brain-computerinterfaces, and/or another type of intermediary device or tool for thedevice-based platform components and/or offline and/or hybrid-typecomponents. The use case for this modality can also be where the user isnot impaired in terms of the user's manual dexterity but where the userrequires the use of their hands for other purposes or functions. Such adevice can include Virtual Reality/Augmented Reality and/or mixedreality training devices such as pop-up displays on visors, helmetsand/or glasses and/or holographic type projections. For some users aTangible User Interface (TUI) may be preferable to the graphical userinterface (and/or the keyboard). Examples of TUI can include a physicalpuzzle piece with or without other embedded sensors (grip strength,grasp, galvanic skin response, pulse) and/or equipped to detect andtrack motion.

In step 1525, the system prompts a professional user to select a skilllevel. In some embodiments, the skill level is a mixture of thesectioning strategy, the number of images, interactivity mix, imagecontent and thresholds to evaluate a new category of users from aclinical research and/or training standpoint. In some embodiments, theskill level is chosen from E1-10 (Easy1-10), M1-10 (Medium 1-10), orH1-10 (High 1-10), where each of E1, E2, E3, . . . E10 are differentskill levels, each of M1, M2, M3, . . . M10 are different skill levelseach of H1, H2, H3, . . . H10 are different skill levels, which labeledsequentially according to increasing or decreasing degrees ofdifficulty. In some embodiments, a 1 is the easier level and a 10 is thehighest level within a skill level bracket.

Leveraging complexity variables as a function of the platform's skilllevels, combined with the “game is not the only assessment” approachgives the platform a powerful capacity and versatility to address amultiplicity of cognitive issues and/or learning and/or trainingsituations. For example, the number of interactive elements (gamepieces); image content; image type, including: photographs, artwork,line drawings, color, and/or halftone, b/w; number of images (2+) usedin a composite; size of the elements; the content character of theelements-high/low detail and color variability; spatial assignment ofthe elements; the presentation mode of the elements-grouped or random,single elements; individual image sectioning strategy, such as: 1:2(50%), 1:4 (25%), 1:6, 1:8, 1:10, 1:12, 1:15, 1:20, and/or higher/lowerpercentages; fixed, mixed or variable sectioning strategy between theimages and within an interactivity; type of interactivity mix for bothcomponent/single images and/or composite images; hint availability anduse of reference images, timed/un-timed interactive components; timeconstraints (time to completion requirements); AI adaptations andtolerance/threshold levels; word lists and number of words to berecalled in an assessment protocol; SQ2 questions, number of questionsand level of difficulty; among other elements can be modulated to meet auser's needs and/or user groups' assessment, learning, remediation,and/or training needs.

In steps 1530-1545, each User (A-D) selects a skill level. For a giveninteractivity, the size and width of an image's sections, together withthe image and/or image set's complexity can be varied to reflect auser's skill level and/or changes to their skill level, and/or abilitiesto make dynamic adjustments to gameboards and game pieces in assessingand challenging cognitive function on a global cognition basis(multi-domain), but with a particular focus on attention and memory, andin general towards evaluating the user's strategy and problem-solvingabilities using the interactivities in terms of user process,solution-finding, task completion and follow-up assessments.

In step 1550, the system assigns the users of the same skill level (E, Mor H) to a user group. For example, in step 1550, User B and User C areput in the same group because User B and User C selected skill level Mand in step 1555, User A and User D are put in the same group becauseUser A and User D selected skill level E. In some embodiments, settingmultiple users to the same group allows the professional users to betteranalyze the progress of a user by comparing that user to other users ofthe same skill level as a cohort. The professional user may alsoidentify whether a user needs to be moved to another skill level basedon this comparison or in other embodiments the system's AI logic mayinitiate recommendations or depending on the configuration automaticallyperform skill level or other adjustments to user interactions with theplatform.

In an embodiment, each of the steps of method 1500 is a distinct step.In another embodiment, although depicted as distinct steps in FIG. 15,step 1505-1555 may not be distinct steps. In other embodiments, method1500 may not have all of the above steps and/or may have other steps inaddition to or instead of those listed above. The steps of method 1500may be performed in another order. Subsets of the steps listed above aspart of method 1500 may be used to form their own method.

FIG. 16 is a second example of a collaborative method in whichprofessional users (e.g, health care workers) analyze data from usersbased on skill levels 1600 (see also FIG. 16). In FIG. 16, professionalUser A and D are collaborating.

In steps 1605 and 1610, Professional User A and D choose a Mem+ (A andD). In step 1615, the system prompts a user to select a skill level.

In steps 1620 and 1625, User A selects general skill level E3, and UserD selects general skill level E1.

In step 1630, users that choose skill levels E1-3 are assigned a usergroup.

In step 1635, the system prompts the User to select protocol parameters,allowing that user to choose how long he or she wants to work on a testor image set with specific contiguity characteristics, how many the usermay want to do, etc.

In steps 1640 and 1645, User A and User D selects an image set,sectioning, interactivities, thresholds, progressions, targets, etc.allowing User A and User D to create a personalized Mem+ protocol thatis still at skill level 1-3. The images and/or image sets can bepresented to the user by the system according to a protocol and/or canbe selected by the user.

In step 1650 and 1660, the system builds a Mem+ protocol based on theskill level and selections (in steps 1640 and 1645) for each of User Aand User D. Progressions refer to the sequence of interactivities. Forexample: staying with “Construct” for two images where the user firstuses 25% sectioning, then 20%, then 10% is a significant skill leveljump, and which lets the user downgrade (or upgrade the skill level) toan in-between point or to a higher skill level (7.5% cuts, 5% cuts) orto 3 images, not just two images. Progressions can also define whatcomprises an Interactivities Set (Compose 25%; Construct 20%; Construct10%; Missing Pieces 4×4 grid; MatchME! 25% cuts with half and quartersize pieces).

In step 1655, a collaboration space allows the Professional users A andD (and perhaps other Professional users) to share protocols and comparedata for group E1-3. Utilizing a collaborative format and forum, aprofessional configuration can be shared and re-purposed and/or modifiedby other Professional users for their applications without having tobuild a protocol from scratch. The original protocol configurationdeveloped by a Professional user would remain unchanged but where otherusers are provided with copies of the configuration for their use. Thecollaboration can be shared protocols for research. Thus, the platformcan be a research platform for collaboration and sharing of protocols.

In steps 1665 and 1675 UserA and UserD Mem+ protocols are deployed tothe end-users. As determined by the administrator, a professionallydeveloped configuration can be deployed in the consumer end-user space.In some embodiments, this results in an output of assessment data to thecollaboration space to build the platform's analytics capabilities witha dynamic source of additional datasets.

In step 1670, the Professional users can access the data center andcreate working group condition spaces in step 1680. As determined by theadministrator, a professionally developed configuration by voluntaryagreement can be tested with different user groups in a modifiedevaluation of the configuration by target audiences. “Condition Spaces”can be defined as areas of collaboration for specific diseases.Specialists tend to think about cognition and the kinds of patients aclinician would be treating in this way by their disease and thusprovides a useful, though limited, perspective on disease-basedcognitive associations

In an embodiment, each of the steps of method 1600 is a distinct step.In another embodiment, although depicted as distinct steps in FIG. 16,step 1605-1680 may not be distinct steps. In other embodiments, method1600 may not have all of the above steps and/or may have other steps inaddition to or instead of those listed above. The steps of method 1600may be performed in another order. Subsets of the steps listed above aspart of method 1600 may be used to form their own method.

FIG. 17 is an example of a method that allows professional users tocreate a cognitive platform 1700 for specific uses (e.g, tests,diagnostics, treatments of specific diseases) in a collaborative way.The cognitive platform can be produced using collaborative processesbetween Professional users.

In FIG. 17, two Professional users (A and D) are collaborating toproduce a cognitive platform for a joint study.

In step 1705 and 1715, Professional user A and D each activate a Mem+protocol.

In step 1710 and 1720, Mem+ protocols are produced for Users A and D.

In step 1725, a collaboration space is produced for sharing protocolsand comparing data. Steps 1730 are sub-steps of an embodiment of Step1725. As discussed in FIG. 16, 1655, in the collaboration space aprofessional configuration can be shared and re-purposed and/or modifiedby other Professional users for their applications. The originalprotocol configuration developed by a user remains unchanged and otherusers are provided with copies of a configuration for their use. Thecollaboration can be shared protocols for research. Thus, the platformcan be a platform for collaboration and sharing of protocols that arebest for a specific patient, group of patients, type of patient, orother user group (see steps 1730-1750): In step 1730, clinicians shareprotocols. In step 1735, clinicians customize protocols based on thesharing. In step 1740, third party image sets are uploaded. The imagesand/or image sets may be presented to the user by the system accordingto a protocol and/or can be selected by the user (e.g., a professionaluser). In step 1745, subjects are enrolled. The subjects may be anyonethat wants be take the customized protocol. In step 1750, forms(questionnaires, surveys and assessments) are prepared, given tosubjects and included in the collaboration space. Protocol userstatistics and associated data, including questionnaires and assessmentsare kept separate and in a research safe assigned to each professionaldeveloper who is conducting research studies so as to protectparticipants' identities and other privacy related data andconsiderations.

In step 1755, working group condition spaces are identified. Workinggroups may include groups that are appropriate for specificcollaborative protocols (e.g., patients with the same diagnosis,patients at the same skill levels, people who want to use the protocolsfor keeping their cognition healthy, patients who need a customizedtreatments those enrolled in a specific test protocol, etc.).

In step 1760, a data center is accessed. The data center may contain theinformation about whether a user can access a specific protocol. Insetting up safeguards ensuring only users authorized to access aparticular set of data have access may help ensure that a protocol maynot be accidentally provided to the wrong user. In this embodiment auser may be a professional collaborator.

In step 1765 final collaborative protocols are released to thecollaboration space to be used by the collaborators.

In some embodiments, steps 1725-1750 can be skipped and the ProfessionalUsers can immediately include their Mem+ protocols within the workinggroup condition spaces. In some embodiments, steps 1725-1755 can beskipped and the Professional Users can immediately include their Mem+protocols in the data center (step 1760). In some embodiments,Professional users can immediately include their Mem+ protocols asreleased products (step 1765).

In an embodiment, each of the steps of method 1700 is a distinct step.In another embodiment, although depicted as distinct steps in FIG. 17,step 1705-1765 may not be distinct steps. In other embodiments, method1700 may not have all of the above steps and/or may have other steps inaddition to or instead of those listed above. The steps of method 1700may be performed in another order. Subsets of the steps listed above aspart of method 1700 may be used to form their own method.

FIG. 18 shows an embodiment of a PICSSi prototype GUI. In someembodiments, the PICSSi platform GUI components can include: a digitalapplication, an expandable image library, categories of interactivitiesor a pre-defined battery of interactivities (Jumble-Sort, Compose,Missing Pieces, MatchMe!) which may be applied to individual images andto 2-image composites (or multi-image composites) at fixed or variablecomplexity levels. The percent (%) sectioning strategy (that is thepercentage of the total image that each sectioned part is) may bevaried, which may affect the number and size of the playing pieces. Inan embodiment, a timer can be toggled and hidden. In an embodiment, usergameplay statistics (alpha-type speed and accuracy assessments) may bedisplayed, and which may include the time taken for each move, the timetaken for each group of moves, reaction time (time to first move),average reaction time, and/or the total time taken to complete theinteractive, number of errors and which may be stored as part of theuser stats and incorporated into built profiles for registered users. AMem+ protocol may designed to be taken in a unit of 12-sessions, inwhich the user performs 2 sessions per week with fixed interactivitiesand image sets, at the easy level, for example.

In another embodiment, the platform may include a Screening/AssessmentTool and/or Therapeutics module, with multiple play options, which mayinclude, but are not limited to, FreePlay: user defined imageselections, interactivities, and complexity; and/or Auto-Sequence Play.Auto-Sequence Play may be a set of interactivities based on userprogress through a pre-defined sequence of increasing complexity, inwhich the user completes tasks within countdown clock and proceedsthrough the pre-set thresholded progressions. The platform may include aCognitive Health Sequence, which may be similar to sequenced play, butwith embedded Mem+ assessment (Remedial, Easy, Medium, Hard levels),with additional scales, and/or to include the use of an optional visibletime. The platform may include specialty modules, which may be a subsetof interactivities and related assessments which may be deployed (e.g.one of the interactivities may include Stroke Recovery module, which mayinclude an interface and configuration personalized for those who havesuffered a stroke, and therefore may include a view-only/auto run, forexample) with a progression from VVO to Hands-on modalities as usersrecover/regain language and/or fine/gross motor abilities.

In some embodiments, there may be alternative presentation modes. In onepresentation mode, there may be a user-specific registration andback-end user data stored, Language pack which has an associationbetween words and elements of images, and/or an expanded library whichexcludes a subset of images based on color, content and/or context. Theexpanded library may include a number of images that are potentiallysearchable based on content tags, User-supplied images, semi-automatedcontiguity analysis, image descriptors tagging, and/or image usagestatistics to eliminate potential bias in user selection and overlyfrequent use of images.

In some embodiments, user statistics may include a movement mapper (thatmaps movement of the user as the user interacts with the interactivity),saved data linked to the user, a Slide bar to adjust percentagesectioning, Unequal sectioning of the different images (e.g., the sizeof the section of image 1 may be different than the size of the sectionsof image 2, which in-turn may be may be different than the size of thesections of Image 3). The presentation may include a tool for convertinga 2-image composite image (e.g., which may have strips of two imagesinterleaved with one another) to a 3-image composite image (e.g., byadding a strip of the third image between a strip of image 1 and a stripof image 2). The presentation may include increased size of thecomposite image as the reference image. The presentation may include anImproved Navigation. The presentation may include an option for fixingor changing the size/change of an image changing the content of animage, a feature for pausing the Game pause, and Piece rotations, amongother presentation features.

FIGS. 19A-D shows a table of rules and values that summarize somecomputations that may be performed to identify and/or characterizecontiguities. Referring to FIG. 19D, the prominence and number ofcontiguities may be represented by a contiguity rating value (CR alsoreferred to as the Ambi Value or juxtaposition value), which may becomputed based on the formula, CR=Σ(AF₁+AF₂+AF₃+AF₄+AF₅+AF₆)/n (wheren=6), where AF₁, AF₂, AF₃, AF₄, AF₅, AF₆ are ambiguity factors (AF). Inother embodiments, there may be other factors and/or one or more of AF₁,AF₂, AF₃, AF₄, AF₅, AF₆ may be divided into multiple factors, while oneor more others of AF₁, AF₂, AF₃, AF₄, AF₅, AF₆ may be left out therebychanging the value of n.

As indicated in the table, AF₁ is a contiguity number, which isdetermined by detecting edges, using an edge detection technique and/orthreshold techniques edge detection technique and/or other types offilters, which produce a binary image based on a threshold thatdetermines which of two values a pixel is assigned.

Contiguity Count Total (AF₁) is the average of the count of contiguitiesbased on a variety of methods of counting contiguities. For example, anumber of different threshold images may be produced for a variety ofintact or different stitched images (which may be stitched differently),where the thresholded values for the image measured at a starting pointof 127 value (for example) and then at 160 (for example) for standardimages, where the color may be represented by pixels values of 0 to 255,for example, and for each image and stitched image the number ofcontiguities are counted. The number of contiguities may also beseparately computed from the edges generated by an edge detectiontechnique, such as a Sobel. A variety of color map images may begenerated for a variety of different stitches, and the contiguities foreach image may also be counted. Then the total number of contiguitiescounted for each variation of the image and method of countingcontiguities are averaged.

More than just two thresholds may be computed. For a thresholded Imageat 127 and 160, Averaged Contiguity Count_(T127)=(Parts_(T127b)+Parts_(T127w), Averaged Contiguity Count_(T160)=(Parts_(T160b)+Parts_(T160w))/2, where Parts_(T127b) and Parts_(T160b) are the numberof parts of the image, that after thresholding have an average pixelvalue of black, and where Parts_(T127w) and Parts_(T160w) are the numberof parts of the image that after thresholding have an average pixelvalue of white, and the subscripts T127 and T160 represent the thresholdused for generating the threshold map. Each part may be a continuousregion of a set of contiguous pixels of the same pixel value afterthresholding. In an embodiment, one may count the number of black andwhite regions across the width of the image to arrive at the number ofparts (e.g., along the central horizontal axis of the image or a long aline that is halfway between the top and the bottom the image). Inanother embodiment, a vertical disruption larger than a predeterminedthreshold divides a region into different parts. Additionally, oralternatively, the horizontal disruptions may also divide a region intoparts. Additionally, or alternatively, disruptions in other directionsmay also divide a region into parts. In an embodiment, a disruption ismore than 50% of the distance from a first edge to a second edge facingthe first edge. For example, a vertical edge that is 50% of the distancefrom the top edge to the bottom edge of the region divides a region intoparts. In other embodiments, the ratio of the length of the disruptionto the distance between the opposite facing edges (e.g., between the topand the bottom edge) may be a different percentage, such as 15%, 25%,75% or 80%.

Contiguity Count Total (AF₁)=(Averaged Contiguity Count_(T127)+ AveragedContiguity Count_(T160))/2. AF₂ is the color block. Color blocks may bedetermined based on a sequential color extraction using a reduced, fixednumber of colors (e.g., 2-6) from which color images may be based. Colorblocks are a kind of contiguity. AF₂-CB defines the distribution ofcolor. A color block may extend in any direction. A color block may beformed by a concentration or density of similar colors representing anobject or region across a continuum or continuous region in both thehorizontal and vertical directions. An example of a color block is thesky. Even in a stitched image, the sky can be blue, albeit of differenthues, across the width of an image. The image may be divided intoregions (e.g., quadrants and sub-quadrants) and dominant color or colorsare determined for each region. Color blocking allows for theidentification and analysis of the colors in an image. Color blockingallows for an analysis of the colors in an image, the distribution ofthe color, and the identification of breaks in the block may bedetermined, indicating the presence of one or more vertical disruptorsor other objects. The interruptions in color confluency can disrupt thecolor block's saliency and/or facilitate identifying what the colorblock is. In this process, the image is progressively reduced to asmaller number of colors (e.g., less than 8, less than 7, less than 6,less than 5, less than 4, less than 3) During color reduction, thepixels may grouped into bins of a histogram according to which color bincolor value of the pixel is closest (e.g., if the image is reduced tothe colors having color pixel values 100 and 200, then a pixel with acolor value of 75 would be place in the bin for the color 100). A colorextraction is performed on each color-reduced image to determine thenumber of pixels in each color bin. The values are averaged to arrive atthe AF₂. Up to 6 color blocks can be identified and used for thecalculation, depending on the number of colors and their percentage ofcontribution to the image.

For example, for a 3-color reduction the formula for the AF₂ is CB.c_(x) (AF₂)=(CB.c2+CB.c3)/3. More generally, the formula for AF₂ is CB.c_(x) (AF₂)=(CB.c2+CB.c3+ . . . CB.c(n))/n (where n is the number ofcolors which are in the image, and which is an integer number having avalue selected form the numbers 2-6). In the above formula CB.c2 is thenumber of regions of contiguous pixels of one color identified after areduction to two colors. CB.c3 is number of regions of contiguous pixelsof the same color identified after a reduction to three colors, andCB.c(n) is the number of regions of contiguous pixels of the same coloridentified after a reduction to n colors.

AF₃, is contiguity linearity (C_(linearity)) for a contiguity using astitched image. AF₃ may be computed from C_(linearity)=C_(A)+C_(D),where C_(A) is a value that represents an average of the degree to whichthe angle of the contiguity changes (e.g., the angularity) across thecontiguity, and C_(D) is average the number of breaks in the contiguity.C_(D) also represents a value that reflects how disrupted the contiguityis, as measured using the stitched image. For example, in an embodiment,C_(D) may have one of two values, which are 0 and −0.25, where C_(D) isassigned the value of zero if the contiguity spans more than 75% of thewidth, and C_(D) is assigned a value of −0.25 if the contiguity spansless than 75% of the width.

The contiguity angle may be computed from C_(A)=(L2C _(x) +R2C _(x) )/2,where L2C is the angle made by a line connecting the center of thecontiguity to the point where the contiguity intersects the left side ofthe image, and R2C is the angle made by a line connecting the center ofthe contiguity to the point where the contiguity intersects the rightside of the image R2C is the line that best approximates the angle madeby the right side of the contiguity whether or not the contiguityintersects the right side of the image. L2C is the line that bestapproximates the angle made by the left side of the contiguity whetheror not the contiguity intersects the left side of the image.

Some rules for determining linearity according to at least oneembodiment are as follows. The values in this discussion are based onthe angle of the dominant contiguity and, the distance off of theX-axis. The measured angles are computed and averaged. The measuredangles are further distilled with rules, so that images which differsignificantly in terms of content can be still be grouped andcategorized according to their angular complexity. However, having theangularity data for each stitch and peel image additionally allows forthe extraction of other information.

A value of 0 is assigned if the contiguity disruption is a straightedge, extending across more than 75% of the image width and if theaveraged angular difference of a single baseline point is less than 5°.

A value of 0.15 is assigned to the linearity if the average angulardifference is between 5° to 30°. A value of 0.25 is assigned to thelinearity if the average angle difference is between 30° to 45°. A valueof 0.75 is assigned to the linearity if the average angle difference isgreater than 45° and if the contiguity extends across the image as adiagonal. A value of −0.15 is assigned to the contiguity if thecontiguity is disrupted and/or non-linear (or irregular). A value of−1.0 is assigned to images without a defined contiguity or without anobject-based contiguity. For example, if the only contiguity is the sky,that contiguity has a linearity of −1.0.

In this embodiment, a solid block of color is not viewed as a horizoncontiguity with linearity. If there is a horizon type of contiguity, thevalue of the horizon contiguity is different than −1, but in thisembodiment, as a color block the sky has no linearity, per se, asdefined by angles or disruptions since there are no disruptions in thesky's continuity.

In an alternative embodiment, the absolute value of the sine of theaverage angle (or the square of the sine of the average angle) may beused for linearity for contiguities with no disruptions.

Referring to FIG. 19A, AF₄ is a continuity value (C_(continuity)) for acontiguity using a stitched image. AF₄ is computed fromC_(continuity)=C_(VD)+C_(IE), where C_(VD) is a value representing theaverage of the span (e.g., average of the total width of all) of thevertical disruptors per contiguity, and Cm is an average of the span ofirregular edges of a contiguity. Some examples of VDs are a tree, agrove of trees, or a house on an otherwise continuous contiguity. EachVD has a height and can extend from the contiguity to the top edge ofthe image or to points in between. The irregular edges (IE) refer towhat can be likened to an uneven surface—a rocky shoreline, or a citylandscape which forms an irregularly edged (uneven, bumpy) contiguity byvirtue of the color block of sky above and the continuity of thebuildings across some or all the horizon.

The Continuity Rules for assigning values to images with VerticalDisruptors and/or Irregular Edges are summarized in FIG. 19A. TheContinuity Rules are: if an image has at least one contiguity which iscontinuous across the entire width of the image (75-100%+/−3%), thenassign a value of 1.0. If the contiguity is continuous across50-75%+/−3% of the image, then 0; if less than 50% or if contiguitynumber is 0, then assign a value of −1.0. If there is/are a verticaldisruptor extending more than 5% but less than 30%, individually or ifcombined, up from an otherwise linear and continuous contiguity butwhich has additional complex contiguities, then assign a value of 0.5.If there are 2-3 VD but which are spatially separated, then assign avalue of 0.5. If the vertical disruptors individually extend in thevertical direction less than 20% of the distance to the top of the imagefrom an otherwise linear contiguity then the VD is assigned a value of0.5.

If there are multiple vertical distractors present in the image (treesin the foreground), then assign a value of −1.0. Optionally one can useprogressive decrements to identify variations/objects off the X-axis andtheir return to an established baseline across the entire image. Ifthere are multiple irregular edges on one or more contiguities or ifthere is a single contiguity without a color block greater than 30% ofthe image's height above the IE, then assign a value of −0.25. Assign avalue of −0.15 for a single contiguity with a poorly defined edge whichmay be interrupted across the width of the image, be irregular, or havevertical disruptions, but which is adjacent to at least one continuouscolor block or a color block greater than 30%.

For Irregular Edges, a poorly defined edge is a contiguity which isirregular, and/or which has multiple vertical disruptions throughout itswidth and/or clustered in regions. From a quantitative standpoint apoorly defined edge would be an edge having multiple Vertical Disruptorspresent along the entire length of the contiguity, disrupting thehorizon interface and/or where less than 30% of the contiguity'sinterface has a discernible color block above the disrupted portion ofthe contiguity. The percentage of disruption may also be defined by aseries of grid tools to evaluate how much space a VD occupies and thecolor block above and around it.

The C_(VD) is computed using the above contiguity rules (FIGS. 19A and19B).

Note that the formula below is used to determine where a VD meets thecriteria for the rules. The formula accounts for multiple verticaldisruptions. For example, for a farmhouse on the prairie with a silo,windmill, barn and house in otherwise open space, each of the elementswould represent a VD which would be analyzed according to each VD'scontribution to the overall VD impact to disrupting the contiguity'scontinuity, because the individual VDs are considered to define the VDrelative to one another (the space between VDs from a width perspective,and the height parameter for the image as defined by the contiguity'sY-location).

To compute the C_(VD), the Sub-area_(dc) is the area above the dominantcontiguity. The distance between vertical distractors is measured. Theratio of the area of the first vertical distractor to the subarea (e.g.quadrant) in which the first vertical distractor is in is computedaccording to the formula:

C _(VD.a1)=Vertical Distractorarea1=(VD1_(Q1w))(VD1_(Q1h))/Sub-area_(dc)

VDmQ_(nw) is the width of vertical disruptor m of quadrant n andVDmQ_(nh) is the height of the vertical distractor m of quadrant n. Forexample, VD1_(Q1w) is the width of vertical disruptors of quadrant 1 andVD1_(Q1h) is the height of the vertical distractors of quadrant 1. thesubarea is the area above the contiguity, and each C_(VD) is thepercentage of the area above the contiguity that is occupied by thevertical distractor. The above continuity rules are applied to the firstvertical distractor based on the area C_(VD.a1).

The ratio of the area of the second vertical distractor to the subarea(e.g. quadrant) in which the second vertical distractor is in iscomputed according to the formula

C _(VD.a2)=Vertical Distractor area2=(VD2_(Q2w))(VD2_(Q2h))/Sub-area_(dc)

The continuity rules of FIGS. 19A-D, are applied to the first verticaldistractor and to the ratio of the area of the second verticaldistractor to the subarea containing the second vertical distractor,C_(VD.a2). The process applied to VD1 and VD2 is repeated for eachvertical distractor C_(VD). In an embodiment, for Irregular Edges, thereis only one definition for a vertical disruptor, which is based on thewidth of the irregularity. (All VD are irregular, but not allirregularities are VDs. For example, the trees of a grove of trees VDsare; the grass or flowers of a field with flowers or grass form an IE orpart of an IE).

C_(IE) describes irregular edges as part of computing the contiguity'scontinuity according to the following rule: If there are multipleirregular edges present on one or more contiguities; or, if a singlecontiguity is present but without a vertically adjacent color block withan area greater than 30% of the image above the contiguity, then assigna value of −0.25. Assign a value of −0.15 if there is only a singlecontiguity with a poorly defined edge, but which is adjacent to at leastone continuous color block, or has a vertically adjacent color blockwith an area greater than 30% of the image, above the contiguity.

Referring to FIG. 19D, AF₅, is the color block depth 100, which definesthe color block distribution (see step 524, above). The assignment of avalue follows a set of rules described in FIG. 19B using aquadrant-based analysis of the color distribution in the image. The FIG.19B rules table applies to both AF₅—Color Block Depth 100 (CBD₁₀₀) andCBD_(ST), for the stitched image.

Referring to FIG. 19D, AF₆ is the spatial color-contiguity, whichcompares the contiguity number to the color block number. To obtain avalue for AF₆, compare the value obtained for AF₂ (Color Block) to AF₁(# of contiguities present in the image), and are summarized in thetable in FIG. 19C as follows: If AF1 is equal to AF2, then assign avalue of 0; if AF1 is greater than AF2, then assign a value of 1; if AF1is less than AF2, unless the contiguity number is equal to 0, thenassign a value of 2; and, if AF1=0 then assign a value of −1.

FIGS. 20A and 20B show the application of a stitch and peel, accordingto at least one embodiment. FIGS. 20A and 20B show two versions of animage of a buck, which includes image 2000 a which is the full imageprior to being, and image 2020 b which is the same image as image 2000 aafter being stitched.

In at least one embodiment, image 2000 a (FIG. 20A) are divided intopredefined sections, e.g., a first section, a second section, and athird section. For example, white lines 2014 a and 2016 a divide image2000 a into three horizontal strips, each horizontal strip is about ⅓ ofthe full image.

In FIG. 20B, the first section can be shifted from being adjacent to asecond area to being adjacent to be a third area, so that the firstsection can mask the second section, thereby removing the second section(which in one embodiment may be ⅓ or 33% of the image) from theresulting image, which is image 2020 b shown in FIG. 20B, so that inFIG. 20B, line 2030 b is in the location where line 2014 a has beenmoved to in the process of removing the middle section of image 2000 a.The locations of line 2030 b is also the location were line 2016 a waslocated in FIG. 20A, So that in image 2020 b, lines 2014 a, 2016 a and2030 b are all in the same location as a result of removing the middlesection of image 2000. The first section can be peeled at anotherpredetermined value, for example, at twenty percent, as represented byimage 2020 b (FIG. 20B) in which the central 25% of the image isremoved); alternatively other amounts (e.g., 6.26%, of the total image,12.5%, of the total image, 18.75% of the total image, etc.). Differentaspects of the contiguities and the images, as a whole are emphasized,and by averaging the characteristics of the different stitched versionsof the same image of the images 2020 a and 2020 b features that may bemissed by looking at the image as a whole may be found or by looking atany one given stitch. In other embodiments the stitching and peeling maybe done in a different way than what is depicted in FIGS. 20A and 20B).For example, a different percentage of the image may be removed (e.g.,90%, 80%, 60%, 40%, or 20% may initially be removed), and differentpercentages may be put back (e.g., the put back parts in each stitch maybe 10% of the amount removed, 15% of the amount removed, 20% of theamount removed, 25% of the amount removed 33% of the amount removed, or50%). The amount put back at each stitch and peel may be different. Forexample, in the first stitch 15% of the amount removed may be returned,and in the second stitch 55% of the amount removed may be returned, andas a result of the first and second stitch is to return a total of15%+55%=70% to focus on different aspects of interest.

FIGS. 20A-F also shows the application of a stitching analysis withquadrant-based color blocking, according to at least one embodiment.Image 2020 (FIG. 20B) shows stitch of a first image 2000 (FIG. 20A),according to at least one embodiment. The first image 2000 (FIG. 20A) isdivided into four quadrants, e.g. first quadrant 2002, second quadrant2006, third quadrant 2004 and fourth quadrant 2008. The four quadrantsare defined by horizontal line 2012 and vertical line 2010 (FIG. 20A).

In stitched image 2020 (FIG. 20B) the quadrants overlap to produce firststitch quadrant 2022, second stitch quadrant 2026, third stitch quadrant2024 and fourth stitch quadrant 2028. The stitched image 2020 can enabledetermining that the color blocks between the quadrants, and in which ofthe quadrants are different and asymmetrical. By bringing together twoareas of the image that are not actually juxtaposed, machine system 101can more easily provide an indication of a disruption in the color blockor the presence of an object (e.g., a vertical distractor) in the image.In FIG. 2020, the stitched image of FIG. 2000, the vertical axis 2010 isreplaced with vertical axis 2030 b, while the horizontal axis 2012 isthe same.

In FIGS. 20A and 20 b, the disruption at the seam may be less than thedisruption of the foreground, which facilitates automaticallyidentifying a contiguity (similar to the way the mind is able to piecetogether the horizon and distinguish the background from theforeground).

FIGS. 21A-C shows an example of the application of the use of colorthresholds to extract contiguities characteristics data for imageanalysis. In FIG. 21A, the first photograph 2100 (of the photographs ofFIGS. 21A-C) shows a full color photograph of a beach scene with afence. This photograph of image 2100 shows a number of contiguities,including the fence, the water/sky interface, and the fence cobblestoneinterface. The second photograph 2105 shows a version of 2100 havingcolors at the blue end of the spectrum removed, which brings out thefence contiguity and the fence cobblestone interface, but shortens theocean water contiguity. The photograph 2110 shows a view of thephotograph 2100 that has colors at the red end of the spectrum removedand creates some stronger contiguities (e.g., the clouds can be arguedto become a contiguity), but makes some of the others less distinct(e.g, the fence, water/sky interface and cobblestone/fence interfacebecome less distinct from each other. However, the pattern of thecobblestones becomes more distinct, without providing new contiguities.

FIGS. 22A-D shows an example of composite images comprised of threeimages and how the presence of contiguities in one or more of the imagesaffect the stability of the image in the ground position at any point intime. FIGS. 22A and 22B are multi-stable image sets. FIGS. 22C and 22Dare stable or fixed, i.e. the image in the ground position remains inthe position. FIGS. 22A-D provide an example of how image sets in acomposite image may embed multiple Gestalt principles such asfigure-ground, completion, and continuation which engage top-downcognition and bottom-up sensory processing as the user virtuallyreassembles the spatially separated image parts into the original imagefrom which the image parts came from.

FIG. 22A shows a composite image with a bird on a tree limb as oneimage, a frozen lake with a crack in the ice and shadow extending acrossthe surface as another, and a foggy lake scene with surrounding treesagainst the sky as the third. The full composite image of the bird on abranch when used as one component image of a 3-image composite, showinghow the contiguity “strength” and sectioning strategy can affect whichof the three images is pushed to the ground position. Since all threeimages each have contiguities depending on where the user is looking,that image will be perceived as occupying the ground position. Thesectioning aspect is not trivial and in many ways counterintuitive. Thegreater the number of the sections (up to a point), the smaller theintervening sections which serve as disruptors and the greater theconfluent capacity of the ground image, i.e. the easier it will be forthe mind to reassemble the ground image.

In FIG. 22A, the tree limb in the first image, the shadow on the snow inthe second image, and the interface between the water and the sky,combined with the sky and water color block size and coherency in thethird image, are each reasonably distinct contiguities although eachvaries in width. Consequently, FIG. 22A is multi-stable, because each ofthe images has a contiguity extending across the image. A compositecomprised of two images, each having contiguities would commonly bereferred to as bi-stable.

FIG. 22B shows the top of a deer with antlers, a lake scene and a fieldwith a cloudy sky. In FIG. 22B, the antlers and the top of the body ofthe deer form an object-type contiguity comprised of a single objectagainst a relatively uniform color block (the field), the interfacebetween the ground and the sky (which is in the middle of the image),and the interfaces between shoreline, lake and sky, and the field andthe cloudy sky, each forming contiguities, which make the imagemulti-stable. One noteworthy aspect of the composite image of FIG. 22Bis that to the middle to upper right of the composite image, the bluewater in the lake and sky interface are dominant, but which become lesssalient on the left portion of the composite. Towards the middle left,however, the deer's antlers framed against the meadow allow that portionof the image to assume the ground position; and the field-cloudy skyimage can become the ground image when the bottom 10-20% of the imagebecomes the focal point of the user's attention, interest or focus.

Thus, ground positioning is dynamic based on contiguity dominance. InFIGS. 22A-B, the image set has three contiguities of differentweights/saliency which are spatially separated—giving it switchcharacteristics as a multi-stable image set. This happens because themind predicts what comes next, infers based on information it has,color, content, context which is why the Gestalt principles ofcontinuation, completion and closure work to fill in the gaps, or inthis case to follow the visual cues, ignoring the distractions (gaps orblank spaces). Contiguities may also be formed by large regions of thesame or similar texture and/or coloring (e.g., the sky, a body of water,field behind the deer).

FIG. 22C shows two different floral scenes, one against a sky and theother without a clear distinction, and an image of mountains against astrong sky background. Only the image of the mountains against the clearsky has distinct contiguities, and so the image is stable and the imagein the ground position fixed, i.e. not multi-stable.

Similarly, FIG. 22D shows a snakeskin, raccoon tracks in the sand, and abird walking along a railing against water. Only the image of the birdon the railing has distinct contiguities, and so FIG. 22D is stable andthe image in the ground position fixed, i.e. not multi-stable. Thevirtual reassembly of the image in the ground is a facilitated process.Maintaining ground positioning for one of the images in a multi-stableimage set has a higher cognitive load in removing the distractingflanking content and to prevent a switch if that is a desired attentiontraining protocol. For FIGS. 22C and 22D, constructing the images infigure position may require a higher degree of cognition, since only oneof the three images has a contiguity uniting the components of thatimage. The additional difference between FIGS. 22C and 22D is the easeof identifying what the figure images are; in FIG. 22C floral content isclearly discernible and recognizable, whereas in FIG. 22D, the snakeskinand raccoon have a more abstract quality making their identificationmore difficult and the image set more ambiguous. Both are still stable,but their complexity differs and as such the interactivity skill levelsusing the image sets differ.

FIGS. 23A-F show examples of two-image composite images in whichcontiguities are removed from one of the images to change thecharacteristics of the composite image. The component image content isthe same as FIG. 23A, a 3-image composite. In FIGS. 23A-F, thesequential removal of contiguities transforms a Multi-stable image setinto a Stable image set. FIGS. 23A and 23F are the same.

FIGS. 24A-C show examples of two-image composite images that showhierarchical relationships of contiguities based on image combinationsin figure-ground positioning. In particular, in FIG. 24A, the image inthe ground position has essentially one contiguity, while the image inthe figure position does not have a discernible contiguity (i.e., thesky is not visible). In FIG. 24B, the figure in the ground position hasa multiple contiguities, whereas the image in the figure position haslarge vertical disruptors (flowers) disrupting the interface between skyand ground. In FIG. 24C, the same image is in the ground position, butthe image in the figure positions does not show any sky. Thus, althougheach of FIGS. 24A-C are stable image sets, the stability of the image inthe ground position in FIG. 24B and FIG. 24C is greater than thestability of the image in the ground position in FIG. 24A, but which ishigher than the image in the figure position of FIGS. 24A and 24C. Thesedifferences allow complex, content-rich image sets to be categorized andranked as part of the cognitive platform. These differences can be usedto convey images sets of varying complexity, with differentcharacteristics and which can be integrated intointeractivities-embedded assessments and training protocols. Thesedifferences require different cognitive demand in discerningdifferences, identifying component parts, and resolving ambiguitieswhich can be exploited in the platform.

FIG. 26 is a flowchart showing an embodiment of the cognitive platform2600. A user can use the platform 2600 to treat, diagnose and/or traincognitive processes across multiple cognitive domains. Method 2600starts with step 2601, in which the characteristics of the images usedare analyzed. Step 2601 does not need to be repeated each time the userwants to interact with an interactivity, but may need to be repeatedevery time a user (e.g., a clinician, someone who desires to improvetheir cognitive abilities, and/or a patient) would like to add an imageto the. Platform 2600 includes a system and method for identifyingcontiguity characteristics in an image and a Mem+ Assessment tool. Auser may interact with Platform 2600 online via a GUI or the StandardMem+ Assessment auto-launch and which can also be run offline withmodifications to trap speed and accuracy data, or utilizing the TUIhybrid model—a tactile prop with digital captures.

Step 2601 includes steps 2602-2608. In step 2602, the user/administrator(whom may be a clinician or non-clinician) uploads one or more images.

Step 2600, step 2610 a has two entry points. The flowchart shows twoentry points one defined by the user (2610 b) and potential override ormerge by an administrator (2602) and the steps of the flowchart thatcome after the two entry points from two paths that flow into the samepoint.

Thus, the process can start at step 2602 when the user/administratoruploads one or more images. Alternatively, the process can start when aregistered user begins to use the cognitive platform by logging in 2610(see FIG. 9 user login).

After the user/administrator uploads one or more images in step 2602,then in step 2604, the platform analyzes contiguity characteristics ofthe unaltered image. As part of step 2604, the image characteristics areidentified and/or quantified. Step 2604 may include determining thenumber of contiguities, the contiguity rating, regularity, the number ofvertical disruptors, the linearity, span, distribution, and/or colorblock depth, for example. Step 2604 may include determining content ofimages, which may be added to content tags. Step 2604 may includesubstep of 2606. In step 2606, the image is stitched and peeled todetermine the image characteristics.

In step 2608, the images are associated with descriptive information.For example, content tags (indication of the content of the image),complexity ratings, aesthetic value, contiguity, and/or switch capacity(which may have been determined in step 2604 and 2606) may be associatedwith the image. After step 2608, the method proceeds to step 2609 a.Alternatively, if the images have already been analyzed and associatedwith labels, the user may start with step 2609 a, In step 2609 a, theuser interacts with one or more interactivities, during which the user'sactivities may be monitored and analyzed, so that the interactivitiespresented to the user may be adjusted according to the user's needs andskill level, for example. Step 2609 a includes steps 2609 b-2636.

Step 2609 a has two entry points, which are steps 2609 b and 2610. Step2609 b follows after step 2608. In other words, if the step 2601 wasimplemented, the next step may be step 2609 b. In step 2609 b, the usermay define what the user believes is their skill level. In step 2609 c,based on the user's input about their skill level, the criteria aredetermined for selecting parameters of images for interactivities and/orinteractivities. When the user enters step 2609 a, via step 2609 b, anadministrator may be logged into the plat form by an administratorand/or may the user set up their own account. Alternatively to step 2609b, the user may enter step 2609 a at step 2610, and at step 2610, theuser logs into the platform.

In step 2610 (which is the second entry point to the platform), aregistered user begins the cognitive platform. After step 2610, in step2612 a determination is made whether the user would like to manuallyselect images for the interactivities or whether the user would like theplatform to automatically select the images according to the user'sneeds as indicated by past assessments, user category norms andpredictive analytics, and/or clinician input.

If it is determined that the user wants to select images, the methodproceeds to step 2614. In step 2614, the image database may be accessed,and images available to the user are presented. In one embodiment, allthe images of the database may be available to the user. In anotherembodiment, only images selected by a clinician for the user and/orpreviously selected by the user as images that the user wants to be ableto select from.

Database 2616 is the database of images available to the user, which maybe accessed during steps 2609 b and/or 2614.

Step 2618 may be part of step 2616. In step 2618, the complexity of theimages may be displayed to inform the user regarding which images theymay wish to select.

In step 2620, the user selects one or more images for one or moreinteractivities. As part of step 2620, the user may determine whetherthe user would like an interactivity with a single intact image or aninteractivity that involves a composite of multiple images. If the userdecides they want an interactivity that is composite of multiple images,the user decides how many images will make up the composite and thenselects that number of images.

Returning to step 2612, if the user decided to have the imagesautomatically selected by the system, then the method proceeds from step2612 to step 2622. In step 2622 the system access the user's data ifthey are returning user or relevant information is contained in theirprofile developed at registration. In step 2624, the user's stored skilllevel and the image database is accessed. Then in step 2626, the methodactivates a complexity modulator, which establishes criteria forselecting an image and set of interactivities based on the skill leveland/or cognitive ability of the user.

After steps 2609 c, 2620, or 2624 (or any time beforehand) the activityassessment 2628 is activated, so as to include the user interactivitywith the platform prior to actually interacting with theinteractivities. The assessment may begin at any point prior to thestart of the first interactivity, the Mem+ assessment includes aquestionnaire of the user's habits and general health, as well as anyrecent changes to their health. In an embodiment, a full questionnaireis only used at registration, while a shorter questionnaire is usedprior to the start of an assessment-linked session. Questions mayinclude, for example, “did you sleep well?” “Have you eaten?” “Havethere been any changes in your medication or health since the last timeyou answered these questions?” “How would you rate youralertness/attention on a scale of 1-5?” “Are you wearing your glasses?”For example, assessment 2628 may be activated as soon as the user logsin, sets up an account, and/or is logged in by a clinician.

After assessment 2628 is activated, in step 2630, an image set isselected (e.g., based on the user's skill level as determined in steps2609 c or 2626 or the users selection of step 2620). Step 2630 may makeuse of composite generator 2632 to combine 2 or more images into onecomposite image set which the user or clinician selected, or the systemautomatically selected the multi-image composite for the user. Forexample, composite generator 2632 generates 2 or 3 image combinations.The composite generator 2632 creates composites based on the skill leveland selected images. The Composite Creator/Generator is a system andprocess where 2 or 3 images are serially sectioned and the image slicesalternately juxtaposed.

In step 2634, an interactivity is created for the user based on theuser's self determined skill level, automatically determined skilllevel, input from a clinician, a group the user is a part of, and/or apreassigned interactivity protocol. The width of the sections may bevaried with each image or be the same for both images or within the sameimage. The range of sectioning is between 1.5%-50 percent. (In someembodiments, to include a Slide bar sectioning 1-50%). In otherembodiments, the Composite Creator/Generator can also be used to developa sectioned substrate where individual images can be printed on thesubstrate or for display on a TUI prop. In the substrate printedversion, the blocks may then be combined to create a 2 or 3-imagecomposite, for example, which may have a fixed width. The printed imagesections may be used for the same interactivities using printedtemplates and game pieces. In other embodiments, the composite generatorincludes a generator of a Tangible User Interface (TUI) Prop, which maybe used to interact with an active surface displaying a sectionedportion of an image. The prop may used to virtually “pick-up” adigitally displayed image section so that it can be manipulated as aphysical, tactile entity in TUI prop form; tactile interactions can beaccomplished using a three dimensional printer (3-D printer) that printsthree dimensional objects, such as puzzle pieces having sensors and/orbar coding on the puzzle pieces/interactivity pieces, so as to detectthe user's placement of the puzzle pieces/interactivity pieces on anactive surface. The width of the section can be varied and the propdisplays the section in whatever size it is.

FIG. 26 shows a flowchart of a method 2600 of interacting with aninteractivity. In step 2602, an interface for an interactivity is shownto the user. In step 2604, a determination is made of the type ofinteractivity to be presented to the user. For example, the user can bepresented with a choice of types of interactivities, a clinician maychoose an interactivity for the user, an automated choice may be made bythe system based on input information and/or assessment informationindicating a type of interactivity would be best for the user, thesystem can make recommendations to a clinician, the user may be part ofa group and the type or interactivity may be based on the group that theuser is part of, and/or a predefined protocol may be assigned to theuser that has a preselected type of interactivity. For example, adetermination may be made whether to present to the user with a FreePlayinteractivity in which the user selects each interactivity prior tointeracting with an interactivity, an automated sequence ofinteractivities in which each interactivity is chosen for the user(e.g., by a pre-established protocol), or a cognitive healthinteractivity may be selected in which each interactivity is selectedfor the purpose of diagnosing and/or treating a cognitive issue.

If it is determined in step 2604, to present a FreePlay mode to theuser, method 2600 proceeds to step 2606, and in step 2606, the FreePlaymode begins.

If it is determined in step 2604, to present an automated sequence ofinteractivities to the user, method 2600 proceeds to step 2608, and instep 2608, the automated sequence of interactivities begins.

If it is determined in step 2604, to present a cognitive health mode tothe user, method 2600 proceeds to step 2610, and in step 2610, thecognitive health interactivities begin. After step 2610, the methodproceeds to step 2612, in which intra-activity data (or alpha data) iscollected while the user interacts with the interactivities. Collectingthe intra-activity data may include recording the total time for aninteractivity, recording the time between each move, recording theerrors in placements of pieces, recording the time each move takes,recording the time between the end of a move and the start of the nextmove, the time to the first move, i.e the reaction time, averagereaction time, and recording decision patterns (e.g., what was thesequence the pieces were placed in the correct location based on color,placement order, location sequence? Was the sequence in which the pieceswere placed based on the location of the image part in the image, suchas by placing first the pieces at the edges in the correct locations andthen placing the other pieces in their correct locations).

Next, in step 2614, post-activity data (or beta data) is collected,which may include collecting data that relates to the interactivityafter the interactivity is finished. For example, a series of questionsmay be presented to the user about the interactivity, for the user torespond to. For example, the user may be asked to recall a word listassociated with the interactivity, the user may be asked SQ2 questions(i.e., what color was the flower, the bird in the image looked most likewhich of the following birds), the user may be asked to recall 5-7objects they previously identified in an image, or to recall 7-10 itemspresent or descriptive of the image they worked with in theinteractivities. The input for the word list recall may be received, viatext, clicking on a list of words or clicking on words which and/or byvoice/verbal response. The user may be asked to differentially provide adescription of only one of the images in a stable or multi-stable imageset, such as a list of the objects in one of the images or a scenecaption, which may be provided by text and/or voice.

Next, in step 2616 biometric metric data (or gamma data) may becollected (which optionally may be collected by a third party). Forexample, the biometric data may include, an analysis of the user'svoice, stressor inputs, an analysis of the user's handwriting (where theanswers to some questions are received, via a hand written response, ananalysis of the user's attempts to try to draw different types offigures, conducting a Single Channel EEG (e.g., while taking assessmentor while performing another assessment, tracking eye movements while theuser is performing the interactivity, mapping hand movements and/orother body movements while the user is performing the interactivity,and/or performing an fMRI while the user was performing aninteractivity.

After or as part of steps 2612-2616, the data is analyzed by alphaassessments 2618, beta assessments 2620 and gamma assessments 2622.

Returning to step 2606 and 2608, after steps 2606 and 2608, the methodproceeds to step 2618 for an alpha assessment.

Next data collector 2624 collects the results of alpha assessments 2618,beta assessments 2620, and/or gamma assessments 2622.

Then data analyzer 2626 analyzes the data from data collector 2624.

In step 2628, a determination is made whether there are signatures ofany cognitive issues and/or cognitive strengths (for example, the systemmay be used for assessment whether a user has unusually good cognitiveskills in one or more cognitive domains, i.e. people with Asperger'swith generally excellent visual spatial abilities; or people withWilliam's Syndrome with generally poor depth perception abilities).

The use of real-world image (pictures) allows for the development ofimage-cued word lists for recall assessments. The word list for Mem+ areimage-cued, i.e., where the word list is derived from the images withwhich the user will perform interactivities for a period of time. Thenumber of words which are image-cued can be varied as can the number ofwords used in the memory recall assessment. In one embodiment, 40% ofthe words in a word list will be image-cued. For example, a person canbe tasked to remember 5 words, 3 of which are image-cued or the user maybe tasked to remember 5 words where all 5 are image-cued. Similarly theuser may be tasked to remember 7-10 words where all or only a portion ofthe words to be recalled are image-cued.

The stored user images are accessed based on the user's stored skilllevel. User data that is stored allows the user to pick up where theuser left off in a previous session. In FreePlay mode, the user cancontinue with the same process or change options (change skill level,images, or interactivities). With training mode to advance their skilllevels, other parameters can be changed. If thresholds are not met(e.g., number of tasks completed within a specified time; increase inthe number of errors, total time increase), then game logic might offera downgrade or encouragement or a hint at solving, or automaticallyadjust the skill level slightly to support continued use at the higherskill level/complexity but at its lower end rather than at the middle orhigher end of a given skill level. For example, if the Skill Level ismeasured as follows: Difficult (Diff) can be broken into Diff0, Diff1,Diff2 . . . Diff(n) where Diff(n) where “n” might be 10. If a user hitsall thresholds for Diff(10) then the user advances similarly within theDifficulty level with additional and more complex tasks, higherthresholds to be met and other complexity modulation adjustments whichcan be made. If the difficulty level is Easy a similar design can beused for E0 where the levels are E0.1 . . . E0.n between E0 and E1. Insome embodiments, the platform can adjust the complexity level withchanges to sectioning, color and image content, number of pieces, sizeof the pieces. In one example, a person is given three puzzle sectionsfor a given image and are tasked with reconstructing the whole imageusing the 3 pieces. Another person, with slightly better or capablecognitive status might be given the same image, the same three piecesinitially for construction, but are then tasked with constructing theimage using pieces of the same width but where there is now a mixture of2-whole sections (2-W) and 1 section which has been divided in half(2-H). The progression might continue on the same day, or another day,with a simple training exercise (which also serves as an assessment)where the user is still working with a 3-part puzzle but the sectionsare now 1 whole section and 2 sections which are divided into halvedparts (or quarter parts) fewer parts, larger parts ⇐|E0|⇒ more puzzleparts, smaller puzzle parts, etc.

The αβγ Mem+ data collection and analyzer shown in step 2624 and step2626 module once activated can capture and analyze data from otherinteractivities including intra-activity Word List Recall (T=0′);Delayed Recall (T=5′); Placement Error; Time/Move; Time between Moves,and post-activity Word List Extended Delay Recall (T=10-15′)(text/voice) SQ2 Questions (spatial, quantitative,qualitative)(text/voice); Object Dimensional Descriptors (text/voice);DescribeIT! (text/voice); ObjectID/OIDm (label/voice), and utilizeembedded, API or 3rd Party tools for voice analysis, handwritinganalysis, stressors, eye tracking, single channel EEG, fMRI and otherbiometerics.

Mem+ Alpha may be assessed during the interactivities, traditional speedand accuracy measures are recorded during game play, along withindividual time records of time taken per move, and time between moves(Mem+ Alpha), and movement mapping of users decisions in arriving atinteractivity solutions, including Placement Error Repeats and patterns.The word list recalls (T=0′, 5′ and 15D, together with other post-gameassessments are used to acquire Mem+ Beta data and which together withMem+ Alpha and/or with or without 3rd party Apps are combined to developa user's Cognitive Profile and their Signatures (an over time measure).

Mem+ Gamma data can be used to further refine Cognitive Signatures, weplan to integrate third-party applications which can also be factored infor intra-activity measurements, including: single/multi-channel EEG,eye tracking, and stress level measures (HR, RR, BP, skin galvanicresponse, pupil dilation), among others which may evolve in theirutility over time.

These metrics, together with intra-activity and post-activityassessments can be used to develop Cognitive Signatures which providemore comprehensive point-in-time and changes-over-time insights into theuser's cognitive status.

If the Cognitive Health Sequence is used the User may choose forthemselves and/or the Mem+ αβ option can be activated for them by thesystem.

Mem+ α assessment data related to the interactivities collects speed andaccuracy data as a default and is correlated to the use mode (Freeplay,Challenge, CognItive Health, etc.

A registered user can elect to review a detailed analyses of data at theend of a session and/or interactivity. The Mem+ data collector collectsdata related to the interactivities and assessments. The data caninclude the a Mem+ standard or the αβ (γ) data. Data is capturedcontinuously. Data from αβ is captured before, during and after α and β.Alpha data is collected during and after at least in terms of WL Recall,Dimensional Descriptors, Object ID/Object ID Memory and SQ2. In general,for those interactivities which require a user to respond verbally orwith input, they can be characterized as beta, if there is a break inthe interactivity, and which have accuracy and speed measures associatedwith them.

The αβγ data is analyzed. For a training protocol, the results of αβγ(intra-activity, post-activity and biometrics data, if available) can beused to determine if the user has reached thresholds for advancement inthe complexity level. Those complexity level changes could include, butare not limited to: changing the sectioning strategy, changing the sizeof pieces to be matched or constructed, increasing the color and/orcontent number of objects in an image (e.g., bird on a branch withuniform color in the background can be less complicated than a bison onthe road with the mountains in the background, and a speed limit roadsign). For baseline assessment purposes, age and health normative datacan be developed for a given set of interactivities, using the sameimages, at a given skill level. Using normative references, otherassessment scales for high and low “outliers” can be developedindividually and across a spectrum for cognition as a whole, allowingthe platform the ability to scale skill levels based on user needs.Spectrum outlier ends might be represented by healthy, superior athletesand great thinkers with superior creative and/or critical skills; and,at the opposite end conditions-based outliers on a cognitive degradationscale (e.g., end-stage Alzheimer's disease). In some embodiments, therewould be other scales where a skill may be superior in one group(visual-spatial for users with Aspergers) but where other domain-basedskills are deficient. In some embodiments, for a person with Williams'syndrome visual spatial and depth perception can be compromised togreater/lesser extents but verbal language and reasoning are excellent.The system can be used to identify “global” markers which demonstrateglobal cognitive engagement (skills and processes), anddomain-referenced skills (local). These efforts can be used to informthe development of a high-impact product group which can authenticallytap into associative cognitive networks across multiple domains,assessing, reinforcing and/or improving on existing skills andprocesses, while at the same time identifying/addressing deficiencies—astreamlined, sensitive global assessment and/or interventional strategy.

The αβ(γ) data is analyzed and used to generate the user's CognitiveSignature which is stored and displayed to the user

The Mem+ standard is assessed. This includes speed and accuracy measures(total time, time per move, reaction time, error rate) which areembedded in the process of doing interactivities. The Mem+ standard canbe stored and displayed.

In an embodiment, each of the steps of method 2600 is a distinct step.In another embodiment, although depicted as distinct steps in FIG. 26,step 2602-2632 may not be distinct steps. In other embodiments, method2600 may not have all of the above steps and/or may have other steps inaddition to or instead of those listed above. The steps of method 2600may be performed in another order. Subsets of the steps listed above aspart of method 2600 may be used to form their own method.

Alternatives and Extensions

In one embodiment of the platform, the images and/or image sets aregamified to generate manipulatable game pieces which the user uses inthe various interactivities. Unlike many traditional types of puzzlinginteractivities, game pieces, e.g., manipulative elements generated for,by, and/or used with the platform do not contain fitted-shaped edgeswith fit specificity. Rather, manipulative elements are produced withonly edges on each of its sides. This aspect requires that the user relyon visual and cognitive cues such as: image content, patterns, horizonlines/contiguities, color contiguity coherency as well as user knowledgeand experience in identifying parts and reassembling puzzle piecesand/or other actions for engaging in other interactive manipulations.

In one embodiment using offline non-digital manipulatives, the sectionedpuzzle pieces can be manufactured with a magnetic bud inserted into thesides of the image sections. This format can allow for kinestheticawareness of the image sections top-bottom orientation to provideimmediate feedback on the placement attempt relative to interactivitiesusing a single image composition and/or a composite. For older adultswith pacemakers where the use of magnets can be problematic as well aswith other users, the platform's offline components can be comprised ofa hybrid electronic gameboard using a TUI prop, which can include atimer element and/or be programmed based on the image sets using a QRand/or barcoding type of reader or other type of sensor which canidentify individual game pieces/image sections and the targetedcompleted image and/or image set to evaluate proper placements and mapuser decision-making movements. In one embodiment, the digital gameboardinterface may include embedded sensors lights and/or other visual and/orauditory cues to indicate proper as well as improper placement ofindividual pieces.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

The platform allows users to engage in the interactivities in aself-directed fashion as well as facilitated, group play and competitiveplay. This is possible in part because of the nature of theinteractivities, where turns can be taken, but also because theinteractivities do not have to be time-dependent even though a timer mayrun in the background but which does not have to be visible to the user.The timer can also be used to limit a person's placement time duringgroup play, where every turn is given a specified amount of time inwhich to complete a given turn/placement. Each of these variations ispossible because interactivities are not time constrained and/or thesolution or task completion does not have to be time-dependent.

These options give the platform significant advantages over othercognitive platforms; specifically, for group and facilitatedinteractions. Group interactions can provide workforce partners with anopportunity for competition and team-building on a larger scale lookingat time to complete a series of tasks. The second type of groupinteraction can provide groups of individuals with opportunities forsocialization as each can take a turn or work cooperatively on arrivingat a puzzle solution as part of a group interactivity.

Facilitated interactions can also be overseen by a therapist during atreatment session and/or be provided by a caregiver as a means ofsocialization, but which can also be used to support cognitive trainingand/or treatment according to a set of protocols. Self-directedinteractivities can be applied to the platform's multiple modes,including: Challenge, Freeplay and Mem+ mode—where the latter describesdefined professional protocols developed for diagnostic, assessment,treatment and training purposes.

In one embodiment, the platform can be used for training purposes inVolume Information Processing mode and used in conjunction with a memoryand/or attention assessment. In this interactive mode, the user isprovided with sections extracted from an image and the user is thentasked to identify scenes or objects the user has observed in the task,identify color and identify the scenes of the object's spatial positionwithin the image. Another interactive can be the presentation ofmultiple image sections from 10 or more images, and then tasking theuser to identify parts of the whole from the collection of images in theMatchMe!-Banner interactivity.

In one embodiment, the platform can be configured to include anadditional variable based on the presentation mode and the additionalvariable can be used to build skill level differences based on how theindividual game pieces/interactive elements are presented to the user.The building skill level differences based on how the individual gamepieces/interactive elements are presented to the user may include thepresentation of all game pieces at the same time where the user is thenrequired to sort through the sections. An alternative presentationformat is where the user is tasked with working with a composite imageset, and where the sections from only one of the component images arepresented to the user for placement. Being given only section from oneof multiple images of a composite image set can be used to vary theskill level complexity for the Jumble-Sort interactivity or at the startof any of the interactivities where one or more game pieces can bepresented to the user as whole sections and/or parts of a whole sectionas half and quarter pieces for example.

In one embodiment, the composite image sets can be used in a view-onlymode and presented to the user as a kind of slideshow with and/orwithout sequences as a mini-movie or video clip, depicting themanipulation of image sections in puzzle-type interactivities. Thepresentation mode can also include the component images as intact imagesand/or a sequence which can show the construction and/or deconstructionof one or more composite image sets from its component images and to itscomponent images, respectively as well as labeling image objects andelements to support language recovery such as may be needed following astroke, traumatic brain injury, concussion and/or with minimallyconscious patients who may benefit from cognitively stimulatingactivities. The slideshow-styled presentation can be viewed on a digitaldevice, including: computer, tablet, phone, TV, monitor/screen, IOTdevice, and/or other type of smart device. Parts of the component imageor images can also include text labels placed on image elements as notedabove. Text labels can be configured as part of a multi-language pack tomake the platform user-friendly to non-native English speakers and/or topeople who have developed language challenges associated with cognitivechanges.

In one embodiment, a user may use the platform in an offline mode usingview-only images in print format and/or through a digital device and ata different point in time and which use a different platform module orcomponent. The platform's offline and device-based interoperability andoverlapping component images and composite image sets allow for thetranslation of data from one device to another, from one mode toanother, from one subset of interactivities to another throughout theuser's engagement with the platform and its components for diagnostic,assessment, treatment, rehabilitation, maintenance and skillsadvancement purposes adjusting to the user's requirements and changes intheir cognitive status.

This kind of versatility represents a significant advance over otherplatforms which are generally relegated to either an offline interactiveor device-based interactive but with little or no crossover between thetwo, and are still focussed on using relatively simple stimuli withsiloed skills. Further, many platforms rely on the use ofneuropsychological assessments which were not designed as treatments. Assuch, the invention represents a paradigm shift in developing sensitivediagnostic and assessment protocols towards identifying cognitivebiomarkers and implementing effective treatment protocols as well aslearning modalities to advance user skills in one or more areas.

The platform is particularly well-suited to people who are recoveringfrom a condition which impacts their cognition within a time frame andwhere functional recovery is possible, and where with earlyinterventions within a system which is responsive to their changingneeds in terms of linguistic challenges, fine motor control, limitedmobility and/or need for facilitated use. These functional requirementseither individually and/or together in part or all can be affected overtime through both natural healing/recovery processes and/orinterventional impacts with transitions to a different functionalcapacity and/or other types of improvements and/or compensatory changesincluding the ability to perform platform-associated tasks at the sametime and/or at a higher skill level independently in a self-directedmode over time from previous facilitated use or VO mode.

In some embodiments, the platform can have applications for peoplefollowing stroke, TBI, stress, depression, MCI, Alzheimer's disease andother dementias as well as other conditions. The platform can be usedwith people as inpatients and/or outpatients in rehabilitation settings,and in long-term care facilities where limited mobility and/or downtimebetween therapies and therapist interactions may occur, and where aself-directed or auto-launched, view-only mode type of interaction canprovide an interventional therapy and/or supplemental treatment modalityearlier in the recovery process.

In one embodiment using a TUI prop, the face of the prop can convey animage segment on its surface. When the TUI prop is brought intoproximity of another TUI and/or separate electronic game board, and/orinterfaces with a computer, tablet, and/or other smart device, theselected image is “transferred” onto the game board/screen. The TUI propcan then display another game piece using an automated process oron-demand process initiated by the user, instructing the TUI prop todisplay the next game piece. The sequence can be random and/or placementcan be facilitated with composites by presenting the user with sectionsfrom only one of the component images for directed placements or a mixedgrouping from two or more images.

With a mixed grouping, the platform can provide the user with aJumble-Sort interactivity, which can include an assessment of the user'sstrategy and be varied in complexity with the number and size of thepieces to be sorted. The Jumble-Sort interactive can be applied tocomponent images in the composites or to non-composited images but wheretwo or more sectioned images are mixed together and the user is taskedto separate the image sections to belonging to Image #1, Image #2 and/orImage #3. In one embodiment of the Jumble-Sort interactivity, the systempresents the user with a mixed grouping of one or more images whichinclude both horizontal and vertical sectioning strategies. The user istasked with separating not only the images but to separate theseaccording to their sectioning strategy.

The Jumble-Sort interactivity can be used as a stand-alone interactivityand/or be combined with another interactivity and scored accordingly forcorrect and incorrect sorting, time and strategy, though the user maynot have to be aware of the timed element.

The user can be provided with a reference image or be tasked with usingother visual cues embedded in the images themselves, including: color,content and context to inform user decisions and/or be provided with atextual description of the target image.

The user's ability to complete a Jumble-Sort task, including sortingaccording to a set of rules, as well as other interactive tasks such asa Compose interactive, and/or changes in their ability to complete tasksincluding error rate and speed as well as other assessment components,whether the users are using individual images and/or composite images,can serve as an index of cognitive change. These measures can be used asa baseline and/or as a change metric. In one embodiment, a user'sability to perform interactivities can improve or regress. The changesin the user's ability to perform a task can regress from being able tocomplete tasks using composite images and component image sections, tonot being able to sort individual component image sections along aspectrum of interactivities and/or other regressions and/orprogressions.

Similarly, other users may have a starting point or baseline where theuser is capable of performing single image tasks, such as composingindividual image puzzles using sections, to not being able to assemblesingle image sections into a coherent image, with or without the use ofa reference image. The placement of an individual along a cognitivecapacity spectrum can also be assessed by changing the number and sizeof the sections. In one embodiment, the same image can be used butduring an assessment that image is sectioned differently, varying thenumber and/or size of the “game” pieces/elements used in theinteractivities.

In monitoring a user's progression and/or regression within a skilllevel or between skills levels, the platform uses a multiplicity ofinternally measured factors and which can be used to signal a change inthe user's cognitive profile and corresponding changes in the level ofinteractivity for rehabilitation, treatment and/or skills developmentand/or learning purposes. The internal measures combined with externalinputs can provide a cogent data steam reflecting multiple cognitivedomain driven interactivities which can enhance our understanding of auser's cognitive status and change tracking. The platform can alsointegrate 3rd party sensors to directly input non-platform based factorsto derive an enhanced cognitive profile. Changes of note can include:consistent/inconsistent time to complete tasks across multiple sessions;an increase/decrease in the amount of time it takes for the user tocomplete a task; change in a metric and/or sub-metric for one or moretasks; an increase/decrease in the number of misplacements;increase/decrease in the use of the reference image for on-demanddisplay; Mem+ associated responses (Word List Recalls and SQ2);deliberate/erratic screen movements; response to altered sectioningstrategy trial; response to altered size of to-be-placed pieces; and/orchange in the number of pieces the user works with and/or thepresentation, e.g., all at once or just one or more at a time;integration of other device data such as derived from a mobile phonetracking gait, other movements and other inputs such as forgottenpassword, passcode, increase in the number of misspelled words orconsistent keystroke errors; changes in health status, prescription drugintake, food intake and sleep status; change in gaze and/or eye trackingacross a composite; eye-tracking combined with ERP and EEG analysis,and/or fMRI data for sustained attention and engagement; body languagecues, increase in frustration, and/or changes in mood. Some of theuser-provided information is collected as part of a pre-sessionquestionnaire which the user is prompted to fill out and can be usedwith the platform in any of its modes.

When a user presents changes in one or more factors with measuredimpacts on their cognitive status and/or skill acquisition abilitiesand/or learning capacity, the platform can adjust the user's skill leveland evaluate how the user adapts to the new interactive regimen and/orprotocol. Changes can include moving a user to a more advanced skilllevel and/or to a lower skill level, and/or switch to an alternateinteractivity, and/or introduce an assessment protocol, and/or withinthe same skill level to improve training and/or treatment efficacy.

In one embodiment, the invention's assessments use a combination of userdata for conducting both “point in time” and “changes over time”analyses. Additional analyses gathers data from multiple user groups inorder to help define and identify potential biomarkers for furtherstudy. A biomarker for a given user group can be used to facilitatediagnoses and implement interventions earlier in a disease process. Theidentification of such non-invasive, cognitive biomarkers canpotentially be used with the platform, and/or be used in conjunctionwith other biomarker identification methods such as Big Data analysisand then used in conjunction with the platform to assess the user'scognitive status as well as with other devices capable of measuringphysiological and neuropsychological inputs, and or capturing data. Assuch, the platform can operate as a diagnostic tool and/or an assessmenttool, and/or treatment tool and/or training tool to identify and monitorchange and/or in an interventional treatment modality.

In some embodiments, audio recordings regardless of whether these areobtained from self-directed and/or facilitated assessments can besubjected to additional analyzes and compiled as part of a Big Dataanalysis of multiple users, and/or used to analyze biometrics' changesof the individual user over time using audio recordings of their voice.Voice change indicators, vocal indicators can provide insight intochanges in the user's cognitive status as reflected in nuanced changesin vocal prosody and/or manifested in other communications, physical,physiological and/or behavioral changes, For example, “vocal prosody mayinclude a composite of supra-segmental acoustic features of speech(e.g., in addition to the lexical, syntactic, and semantic content ofsignals). Primary features may include fundamental frequency (F0), whichmay be perceived as pitch and/or intensity, which may be perceived asloudness; and timing, which is perceived as speech rate, rhythm, andpatterning in normal conversation. Related features include jitter andshimmer (cycle-to-cycle variation in frequency and intensity), energydistribution among formants, and cepstral features.” (See: Cohen A S,Dinzeo T J, Donovan N J, Brown C E, Morrison S C. “Vocal acousticanalysis as a biometric indicator of information processing:Implications for neurological and psychiatric disorders.” Psychiatryresearch.2015;226(1):235-241.doi:10.1016/j.psychres.2014a.12.054.Schuller B, BatlinerA, Steidl S, Seppi D. Recognising realistic emotions and affect inspeech: State of the art and lessons learnt from the first challenge.Speech and Communication. 2010; 53(9-10):10621087.)

Each embodiment disclosed herein may be used or otherwise combined withany of the other embodiments disclosed. Any element of any embodimentmay be used in any embodiment.

Although the invention has been described with reference to specificembodiments, it will be understood by those skilled in the art thatvarious changes may be made and equivalents may be substituted forelements thereof without departing from the true spirit and scope of theinvention. In addition, modifications may be made without departing fromthe essential teachings of the invention.

1. A method comprising: sending, by a machine, one or moreinteractivities to a user, the one or more interactivities including atleast one image, the one or more interactivities including at leastrequesting image input indicative of identifying a portion of the image,the machine including a processor system having one or more processorand a memory system; receiving, at the machine, the input from the user;and assessing the input of the user, by the machine, and based on theassessment determining a score that is indicative of cognitive abilitiesof the user.
 2. The method of claim 1 further comprising adjusting theinteractivities, as the user is engaged with a platform using themethod, during a session in which the assessment occurred.
 3. The methodof claim 1, wherein the assessing is based on information gatheredduring a session including one or more interactivities, whereinformation is gathered prior to the one or more interactivities, whenno interactivity is running.
 4. The method of claim 1, wherein theassessing is based on information gathered during one or moreinteractivities.
 5. The method of claim 1, wherein the assessing isbased on information gathered during a session including one or moreinteractivities, after the one or more interactivities when nointeractivity is running.
 6. The method of claim 1, wherein theassessing is based on information gathered that characterizes biometricdata.
 7. The method of claim 1 further comprising receiving input frommultiple clinicians related to an interactivity protocol.
 8. The methodof claim 7 further comprising creating the interactivity protocol basedon the input from the multiple clinicians.
 9. The method of claim 1further comprising creating an interactivity based on the assessing ofthe interactivity including at least one image that is a composite oftwo or more images.
 10. The method of claim 9, the assessing includingan assessment based on a word list associated with the composite image.11. The method of claim 10, the assessing including presenting a wordlist to a user, presenting a distraction to the user, and requesting theuser to recall the word list after the distraction.
 12. The method ofclaim 1 further comprising: determining, by a machine, one or morecontiguities of an image, the contiguity being a group of pictureelements that are adjacent to one another that form a continuous imageelement that extends at least as much horizontally as vertically andthat extends horizontally across most of the image; determining, by theprocessor system, a value that represents characteristics of the one ormore contiguities that are in the image; and determining, by themachine, which images to include in an interactivity based on the valueassociated with aspects of the contiguities and a skill level of theuser.
 13. The method of claim 1, the input being received via a tangibleuser interface.
 14. The method of claim 1, the assessment being based ona decision pattern.
 15. The method of claim 1, the assessment beingbased on errors in placements of element of an image, wherereconstructing at least a portion of an image.
 16. The method of claim1, the assessment being based on reaction time.
 17. A system comprising:a processor system having one or more processor, and a memory system,the memory system storing one or more machine instructions, which whenimplemented, causes the system to implement including at least, sending,by the system, one or more interactivities to a user, the one or moreinteractivities including at least one image, the one or moreinteractivities including at least requesting image input indicative ofidentifying a portion of the image; receiving, at the system, the inputfrom the user; and assessing the input of the user, by the system, andbased on the assessment determining a score that is indicative ofcognitive abilities of the user.